[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260317 16:16:00.920920 16879 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.123.254:44199
I20260317 16:16:00.922679 16879 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 16:16:00.923856 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:00.944648 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:00.944696 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:00.944890 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:00.948113 16879 server_base.cc:1060] running on GCE node
I20260317 16:16:02.332422 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:02.332741 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:02.333122 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764162332938 us; error 0 us; skew 500 ppm
I20260317 16:16:02.334210 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:02.341403 16879 webserver.cc:492] Webserver started at http://127.16.123.254:34087/ using document root <none> and password file <none>
I20260317 16:16:02.342656 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:02.342947 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:02.343386 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:02.349670 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/master-0-root/instance:
uuid: "695516ce5dd34707b37fb75a0b69e90d"
format_stamp: "Formatted at 2026-03-17 16:16:02 on dist-test-slave-kqc6"
I20260317 16:16:02.360941 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.012s sys 0.000s
I20260317 16:16:02.367540 16894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:02.369288 16879 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260317 16:16:02.369746 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/master-0-root
uuid: "695516ce5dd34707b37fb75a0b69e90d"
format_stamp: "Formatted at 2026-03-17 16:16:02 on dist-test-slave-kqc6"
I20260317 16:16:02.370108 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:02.462680 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:02.464537 16879 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 16:16:02.465072 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:02.562284 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.254:44199
I20260317 16:16:02.562556 16945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.254:44199 every 8 connection(s)
I20260317 16:16:02.572597 16946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:02.598616 16946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d: Bootstrap starting.
I20260317 16:16:02.607964 16946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:02.610489 16946 log.cc:826] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d: Log is configured to *not* fsync() on all Append() calls
I20260317 16:16:02.616649 16946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d: No bootstrap required, opened a new log
I20260317 16:16:02.645462 16946 raft_consensus.cc:359] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695516ce5dd34707b37fb75a0b69e90d" member_type: VOTER }
I20260317 16:16:02.646076 16946 raft_consensus.cc:385] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:02.646453 16946 raft_consensus.cc:740] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695516ce5dd34707b37fb75a0b69e90d, State: Initialized, Role: FOLLOWER
I20260317 16:16:02.647794 16946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695516ce5dd34707b37fb75a0b69e90d" member_type: VOTER }
I20260317 16:16:02.648397 16946 raft_consensus.cc:399] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 16:16:02.648741 16946 raft_consensus.cc:493] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 16:16:02.649129 16946 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:02.653586 16946 raft_consensus.cc:515] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695516ce5dd34707b37fb75a0b69e90d" member_type: VOTER }
I20260317 16:16:02.654477 16946 leader_election.cc:304] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695516ce5dd34707b37fb75a0b69e90d; no voters:
I20260317 16:16:02.656522 16946 leader_election.cc:290] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 16:16:02.657126 16949 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:02.659781 16949 raft_consensus.cc:697] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 1 LEADER]: Becoming Leader. State: Replica: 695516ce5dd34707b37fb75a0b69e90d, State: Running, Role: LEADER
I20260317 16:16:02.661459 16949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695516ce5dd34707b37fb75a0b69e90d" member_type: VOTER }
I20260317 16:16:02.662669 16946 sys_catalog.cc:565] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [sys.catalog]: configured and running, proceeding with master startup.
I20260317 16:16:02.673384 16951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695516ce5dd34707b37fb75a0b69e90d. Latest consensus state: current_term: 1 leader_uuid: "695516ce5dd34707b37fb75a0b69e90d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695516ce5dd34707b37fb75a0b69e90d" member_type: VOTER } }
I20260317 16:16:02.673513 16950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695516ce5dd34707b37fb75a0b69e90d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695516ce5dd34707b37fb75a0b69e90d" member_type: VOTER } }
I20260317 16:16:02.674228 16951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [sys.catalog]: This master's current role is: LEADER
I20260317 16:16:02.674543 16950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [sys.catalog]: This master's current role is: LEADER
I20260317 16:16:02.679456 16957 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 16:16:02.695631 16957 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 16:16:02.697623 16879 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 16:16:02.717970 16957 catalog_manager.cc:1357] Generated new cluster ID: 1f566d0b1e1644d5bc0abe686ad07dde
I20260317 16:16:02.718384 16957 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 16:16:02.734863 16957 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 16:16:02.736907 16957 catalog_manager.cc:1514] Loading token signing keys...
I20260317 16:16:02.760152 16957 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d: Generated new TSK 0
I20260317 16:16:02.761495 16957 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 16:16:02.830608 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:02.841768 16968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:02.845132 16969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:02.850118 16879 server_base.cc:1060] running on GCE node
W20260317 16:16:02.850361 16971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:02.851430 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:02.851681 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:02.851877 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764162851849 us; error 0 us; skew 500 ppm
I20260317 16:16:02.852697 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:02.858924 16879 webserver.cc:492] Webserver started at http://127.16.123.193:41193/ using document root <none> and password file <none>
I20260317 16:16:02.859575 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:02.859817 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:02.860131 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:02.861342 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-0-root/instance:
uuid: "7365fecf4e9e48b084df0c90eff83908"
format_stamp: "Formatted at 2026-03-17 16:16:02 on dist-test-slave-kqc6"
I20260317 16:16:02.869081 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260317 16:16:02.875784 16976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:02.877171 16879 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260317 16:16:02.877575 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-0-root
uuid: "7365fecf4e9e48b084df0c90eff83908"
format_stamp: "Formatted at 2026-03-17 16:16:02 on dist-test-slave-kqc6"
I20260317 16:16:02.877907 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:02.906495 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:02.908651 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:02.911846 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:16:02.918704 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:16:02.919049 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260317 16:16:02.919277 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:16:02.919510 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.054617 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.193:33887
I20260317 16:16:03.054698 17038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.193:33887 every 8 connection(s)
I20260317 16:16:03.065171 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:03.078351 17043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:03.086079 17044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:03.107465 16879 server_base.cc:1060] running on GCE node
W20260317 16:16:03.108765 17047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:03.112152 17039 heartbeater.cc:344] Connected to a master server at 127.16.123.254:44199
I20260317 16:16:03.112339 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:03.112592 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:03.112823 17039 heartbeater.cc:461] Registering TS with master...
I20260317 16:16:03.113948 17039 heartbeater.cc:507] Master 127.16.123.254:44199 requested a full tablet report, sending...
I20260317 16:16:03.112784 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764163112766 us; error 0 us; skew 500 ppm
I20260317 16:16:03.116269 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:03.118498 16911 ts_manager.cc:194] Registered new tserver with Master: 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887)
I20260317 16:16:03.120072 16879 webserver.cc:492] Webserver started at http://127.16.123.194:35829/ using document root <none> and password file <none>
I20260317 16:16:03.120813 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:03.121081 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:03.121431 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:03.121440 16911 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35548
I20260317 16:16:03.123865 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-1-root/instance:
uuid: "c0bfe88236c24bc2971392e273683fd8"
format_stamp: "Formatted at 2026-03-17 16:16:03 on dist-test-slave-kqc6"
I20260317 16:16:03.131687 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260317 16:16:03.138051 17051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.139853 16879 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260317 16:16:03.140313 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-1-root
uuid: "c0bfe88236c24bc2971392e273683fd8"
format_stamp: "Formatted at 2026-03-17 16:16:03 on dist-test-slave-kqc6"
I20260317 16:16:03.140707 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:03.170310 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:03.172602 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:03.174499 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:16:03.177712 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:16:03.177987 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.178284 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:16:03.178491 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.277576 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.194:34563
I20260317 16:16:03.277678 17113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.194:34563 every 8 connection(s)
I20260317 16:16:03.285825 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:03.296434 17117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:03.299573 17114 heartbeater.cc:344] Connected to a master server at 127.16.123.254:44199
I20260317 16:16:03.300053 17114 heartbeater.cc:461] Registering TS with master...
W20260317 16:16:03.300880 17118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:03.301395 17114 heartbeater.cc:507] Master 127.16.123.254:44199 requested a full tablet report, sending...
I20260317 16:16:03.303933 16911 ts_manager.cc:194] Registered new tserver with Master: c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:03.307457 16911 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35552
W20260317 16:16:03.313648 17120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:03.314131 16879 server_base.cc:1060] running on GCE node
I20260317 16:16:03.315126 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:03.315390 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:03.315564 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764163315544 us; error 0 us; skew 500 ppm
I20260317 16:16:03.316097 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:03.318727 16879 webserver.cc:492] Webserver started at http://127.16.123.195:37401/ using document root <none> and password file <none>
I20260317 16:16:03.319284 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:03.319495 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:03.319762 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:03.320808 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-2-root/instance:
uuid: "dfb44a013db548e593d221916ed0dbd7"
format_stamp: "Formatted at 2026-03-17 16:16:03 on dist-test-slave-kqc6"
I20260317 16:16:03.325608 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260317 16:16:03.329870 17125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.330894 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260317 16:16:03.331266 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-2-root
uuid: "dfb44a013db548e593d221916ed0dbd7"
format_stamp: "Formatted at 2026-03-17 16:16:03 on dist-test-slave-kqc6"
I20260317 16:16:03.331586 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1773764160731778-16879-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:03.355643 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:03.356942 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:03.359720 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:16:03.362720 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:16:03.363020 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.363286 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:16:03.363484 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:03.473143 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.195:37361
I20260317 16:16:03.473346 17187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.195:37361 every 8 connection(s)
I20260317 16:16:03.491961 17188 heartbeater.cc:344] Connected to a master server at 127.16.123.254:44199
I20260317 16:16:03.492449 17188 heartbeater.cc:461] Registering TS with master...
I20260317 16:16:03.493278 17188 heartbeater.cc:507] Master 127.16.123.254:44199 requested a full tablet report, sending...
I20260317 16:16:03.495394 16911 ts_manager.cc:194] Registered new tserver with Master: dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:03.496119 16879 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017671236s
I20260317 16:16:03.496812 16911 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38740
I20260317 16:16:03.558104 16911 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38742:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 16:16:03.562175 16911 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 16:16:03.684815 16959 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:16:03.705582 17151 tablet_service.cc:1511] Processing CreateTablet for tablet 995dacb7e66f4c47842e9eba5645d8c6 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260317 16:16:03.704849 17153 tablet_service.cc:1511] Processing CreateTablet for tablet 94613bfab95744bca18c8c52f109387c (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260317 16:16:03.704346 17152 tablet_service.cc:1511] Processing CreateTablet for tablet 7d52fb95165d493db57d26439ab3b47d (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260317 16:16:03.709457 17150 tablet_service.cc:1511] Processing CreateTablet for tablet ae772e8721354fe2bffffe3b986eaf8e (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260317 16:16:03.710685 17151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995dacb7e66f4c47842e9eba5645d8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.715163 17149 tablet_service.cc:1511] Processing CreateTablet for tablet 9527b12d0cba460ea05b4ddcf46a0568 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260317 16:16:03.716382 17148 tablet_service.cc:1511] Processing CreateTablet for tablet 4e2510c1001e4134bf6f61fa639c3551 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260317 16:16:03.718655 17153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94613bfab95744bca18c8c52f109387c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.719534 17152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d52fb95165d493db57d26439ab3b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.720679 17150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae772e8721354fe2bffffe3b986eaf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.725484 17148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2510c1001e4134bf6f61fa639c3551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.728891 17149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9527b12d0cba460ea05b4ddcf46a0568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.732383 17147 tablet_service.cc:1511] Processing CreateTablet for tablet 16a83877f56c4ad7be4fdfb90a68005e (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260317 16:16:03.735529 17147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a83877f56c4ad7be4fdfb90a68005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.751334 17000 tablet_service.cc:1511] Processing CreateTablet for tablet 9527b12d0cba460ea05b4ddcf46a0568 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260317 16:16:03.755942 17001 tablet_service.cc:1511] Processing CreateTablet for tablet ae772e8721354fe2bffffe3b986eaf8e (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260317 16:16:03.758312 17001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae772e8721354fe2bffffe3b986eaf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.760455 17002 tablet_service.cc:1511] Processing CreateTablet for tablet 995dacb7e66f4c47842e9eba5645d8c6 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260317 16:16:03.762449 17002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995dacb7e66f4c47842e9eba5645d8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.763913 17003 tablet_service.cc:1511] Processing CreateTablet for tablet 7d52fb95165d493db57d26439ab3b47d (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260317 16:16:03.764050 17004 tablet_service.cc:1511] Processing CreateTablet for tablet 94613bfab95744bca18c8c52f109387c (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260317 16:16:03.768002 16999 tablet_service.cc:1511] Processing CreateTablet for tablet 4e2510c1001e4134bf6f61fa639c3551 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260317 16:16:03.779829 16998 tablet_service.cc:1511] Processing CreateTablet for tablet 16a83877f56c4ad7be4fdfb90a68005e (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260317 16:16:03.777792 17000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9527b12d0cba460ea05b4ddcf46a0568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.783506 16997 tablet_service.cc:1511] Processing CreateTablet for tablet aff0d35de1f84b6d9e26764809b83e10 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260317 16:16:03.791240 17003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d52fb95165d493db57d26439ab3b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.799719 16999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2510c1001e4134bf6f61fa639c3551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.806849 17079 tablet_service.cc:1511] Processing CreateTablet for tablet 94613bfab95744bca18c8c52f109387c (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260317 16:16:03.809798 16998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a83877f56c4ad7be4fdfb90a68005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.827052 17078 tablet_service.cc:1511] Processing CreateTablet for tablet 7d52fb95165d493db57d26439ab3b47d (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260317 16:16:03.834864 17077 tablet_service.cc:1511] Processing CreateTablet for tablet 995dacb7e66f4c47842e9eba5645d8c6 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260317 16:16:03.848862 17004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94613bfab95744bca18c8c52f109387c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.848958 17079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94613bfab95744bca18c8c52f109387c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.857527 17076 tablet_service.cc:1511] Processing CreateTablet for tablet ae772e8721354fe2bffffe3b986eaf8e (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260317 16:16:03.859445 17075 tablet_service.cc:1511] Processing CreateTablet for tablet 9527b12d0cba460ea05b4ddcf46a0568 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260317 16:16:03.861266 16997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff0d35de1f84b6d9e26764809b83e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.866830 17077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995dacb7e66f4c47842e9eba5645d8c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.874718 17146 tablet_service.cc:1511] Processing CreateTablet for tablet aff0d35de1f84b6d9e26764809b83e10 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260317 16:16:03.876713 17073 tablet_service.cc:1511] Processing CreateTablet for tablet 16a83877f56c4ad7be4fdfb90a68005e (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260317 16:16:03.876673 17074 tablet_service.cc:1511] Processing CreateTablet for tablet 4e2510c1001e4134bf6f61fa639c3551 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260317 16:16:03.888759 17075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9527b12d0cba460ea05b4ddcf46a0568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.893010 17074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2510c1001e4134bf6f61fa639c3551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.885071 17072 tablet_service.cc:1511] Processing CreateTablet for tablet aff0d35de1f84b6d9e26764809b83e10 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260317 16:16:03.889443 17146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff0d35de1f84b6d9e26764809b83e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.898536 16996 tablet_service.cc:1511] Processing CreateTablet for tablet cde56c3fe80f4f94a1981c90ca3eb089 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260317 16:16:03.907307 16996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde56c3fe80f4f94a1981c90ca3eb089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.919067 17078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d52fb95165d493db57d26439ab3b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.942567 17072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff0d35de1f84b6d9e26764809b83e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.962371 17076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae772e8721354fe2bffffe3b986eaf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.965152 17073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a83877f56c4ad7be4fdfb90a68005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:03.999425 17071 tablet_service.cc:1511] Processing CreateTablet for tablet cde56c3fe80f4f94a1981c90ca3eb089 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260317 16:16:04.000893 17071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde56c3fe80f4f94a1981c90ca3eb089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:04.031683 17206 tablet_bootstrap.cc:492] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.038303 17207 tablet_bootstrap.cc:492] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.044168 17149 tablet_service.cc:1511] Processing CreateTablet for tablet cde56c3fe80f4f94a1981c90ca3eb089 (DEFAULT_TABLE table=test-workload [id=2d1f91a7cbd0411d8cb1e65091af146c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260317 16:16:04.048638 17149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde56c3fe80f4f94a1981c90ca3eb089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:04.090649 17206 tablet_bootstrap.cc:654] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.093364 17208 tablet_bootstrap.cc:492] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.105443 17207 tablet_bootstrap.cc:654] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.127238 17039 heartbeater.cc:499] Master 127.16.123.254:44199 was elected leader, sending a full tablet report...
I20260317 16:16:04.130214 17208 tablet_bootstrap.cc:654] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.133260 17206 tablet_bootstrap.cc:492] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.135018 17206 ts_tablet_manager.cc:1403] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.012s
I20260317 16:16:04.141661 17206 raft_consensus.cc:359] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.142805 17206 raft_consensus.cc:385] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.143343 17206 raft_consensus.cc:740] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.144973 17206 consensus_queue.cc:260] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.150485 17207 tablet_bootstrap.cc:492] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.153672 17207 ts_tablet_manager.cc:1403] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.127s user 0.005s sys 0.012s
I20260317 16:16:04.152997 17208 tablet_bootstrap.cc:492] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.155284 17208 ts_tablet_manager.cc:1403] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.014s
I20260317 16:16:04.159307 17208 raft_consensus.cc:359] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.162959 17208 raft_consensus.cc:385] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.163308 17208 raft_consensus.cc:740] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.163520 17207 raft_consensus.cc:359] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.164860 17207 raft_consensus.cc:385] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.165316 17207 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.164036 17208 consensus_queue.cc:260] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.167080 17206 ts_tablet_manager.cc:1434] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.032s user 0.017s sys 0.009s
I20260317 16:16:04.166939 17207 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.168099 17206 tablet_bootstrap.cc:492] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.176111 17207 ts_tablet_manager.cc:1434] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.022s user 0.006s sys 0.005s
I20260317 16:16:04.176340 17206 tablet_bootstrap.cc:654] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.176856 17188 heartbeater.cc:499] Master 127.16.123.254:44199 was elected leader, sending a full tablet report...
I20260317 16:16:04.179566 17208 ts_tablet_manager.cc:1434] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.024s user 0.008s sys 0.003s
I20260317 16:16:04.180843 17208 tablet_bootstrap.cc:492] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.178267 17114 heartbeater.cc:499] Master 127.16.123.254:44199 was elected leader, sending a full tablet report...
I20260317 16:16:04.188441 17208 tablet_bootstrap.cc:654] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.189311 17213 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.190796 17213 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.193254 17207 tablet_bootstrap.cc:492] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.195652 17213 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.203382 17206 tablet_bootstrap.cc:492] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.204186 17206 ts_tablet_manager.cc:1403] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.005s
I20260317 16:16:04.207836 17206 raft_consensus.cc:359] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.208920 17206 raft_consensus.cc:385] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.209298 17206 raft_consensus.cc:740] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.210254 17206 consensus_queue.cc:260] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.214282 17206 ts_tablet_manager.cc:1434] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260317 16:16:04.218349 17206 tablet_bootstrap.cc:492] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.221104 17208 tablet_bootstrap.cc:492] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.222920 17208 ts_tablet_manager.cc:1403] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.000s
W20260317 16:16:04.232589 17189 tablet.cc:2379] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 16:16:04.233052 17208 raft_consensus.cc:359] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.236644 17208 raft_consensus.cc:385] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.237844 17208 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.245769 17207 tablet_bootstrap.cc:654] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.249640 17206 tablet_bootstrap.cc:654] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.243423 17208 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.257925 17208 ts_tablet_manager.cc:1434] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.034s user 0.010s sys 0.000s
I20260317 16:16:04.259202 17208 tablet_bootstrap.cc:492] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.268121 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
W20260317 16:16:04.271225 17054 leader_election.cc:343] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.271657 17208 tablet_bootstrap.cc:654] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.274497 17212 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.275300 17212 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.282233 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.283888 17206 tablet_bootstrap.cc:492] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.286335 17206 ts_tablet_manager.cc:1403] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.000s
W20260317 16:16:04.291837 17053 leader_election.cc:343] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.285226 17212 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.292312 17053 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.296458 17213 raft_consensus.cc:2749] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:04.301671 17207 tablet_bootstrap.cc:492] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.297130 17206 raft_consensus.cc:359] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.303301 17207 ts_tablet_manager.cc:1403] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.110s user 0.011s sys 0.013s
I20260317 16:16:04.303261 17206 raft_consensus.cc:385] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.304740 17206 raft_consensus.cc:740] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.305476 17206 consensus_queue.cc:260] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.310717 17206 ts_tablet_manager.cc:1434] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.019s user 0.004s sys 0.005s
I20260317 16:16:04.311769 17206 tablet_bootstrap.cc:492] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.314673 17207 raft_consensus.cc:359] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.319734 17207 raft_consensus.cc:385] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.320124 17207 raft_consensus.cc:740] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.320931 17207 consensus_queue.cc:260] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.326419 17208 tablet_bootstrap.cc:492] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.326974 17208 ts_tablet_manager.cc:1403] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.068s user 0.022s sys 0.002s
I20260317 16:16:04.328332 17207 ts_tablet_manager.cc:1434] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260317 16:16:04.329984 17208 raft_consensus.cc:359] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.330510 17207 tablet_bootstrap.cc:492] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.333299 17089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.330740 17208 raft_consensus.cc:385] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.334710 17208 raft_consensus.cc:740] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
W20260317 16:16:04.336305 16980 leader_election.cc:343] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.335866 17206 tablet_bootstrap.cc:654] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.336999 17208 consensus_queue.cc:260] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.339195 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.340739 17208 ts_tablet_manager.cc:1434] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.013s user 0.001s sys 0.006s
W20260317 16:16:04.341616 16978 leader_election.cc:343] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.342372 17208 tablet_bootstrap.cc:492] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.342224 16978 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.343369 17212 raft_consensus.cc:2749] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:04.348074 17207 tablet_bootstrap.cc:654] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.349881 17208 tablet_bootstrap.cc:654] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.362223 17208 tablet_bootstrap.cc:492] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.362614 17207 tablet_bootstrap.cc:492] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.362744 17208 ts_tablet_manager.cc:1403] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.003s
I20260317 16:16:04.363092 17207 ts_tablet_manager.cc:1403] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.003s
I20260317 16:16:04.366371 17208 raft_consensus.cc:359] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.366531 17207 raft_consensus.cc:359] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.367275 17208 raft_consensus.cc:385] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.367720 17208 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.367316 17207 raft_consensus.cc:385] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.368510 17207 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.368556 17208 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.369324 17207 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.371042 17208 ts_tablet_manager.cc:1434] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260317 16:16:04.371717 17207 ts_tablet_manager.cc:1434] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260317 16:16:04.372493 17207 tablet_bootstrap.cc:492] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.373584 17212 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.373911 17212 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.375650 17212 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.376784 17089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.377166 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.377650 17089 raft_consensus.cc:2468] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 0.
W20260317 16:16:04.378707 16978 leader_election.cc:343] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.379398 16980 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters: dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.379765 17208 tablet_bootstrap.cc:492] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.380666 17212 raft_consensus.cc:2804] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.381058 17212 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.381772 17212 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.388027 17208 tablet_bootstrap.cc:654] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.397962 17212 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.401762 17206 tablet_bootstrap.cc:492] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.402235 17206 ts_tablet_manager.cc:1403] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.013s
I20260317 16:16:04.402114 17089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:04.402730 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:04.403064 17089 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:16:04.404095 16978 leader_election.cc:343] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.404654 17212 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.405189 17206 raft_consensus.cc:359] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.405666 17207 tablet_bootstrap.cc:654] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.405896 17206 raft_consensus.cc:385] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.406195 17206 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.406936 17206 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.409240 17206 ts_tablet_manager.cc:1434] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:16:04.411353 17089 raft_consensus.cc:2468] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 1.
I20260317 16:16:04.411839 17206 tablet_bootstrap.cc:492] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.412845 17213 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.415403 17213 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.413002 16980 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters: dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.424209 17212 raft_consensus.cc:2804] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.428469 17212 raft_consensus.cc:697] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:04.427042 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.430526 17014 raft_consensus.cc:2468] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 0.
I20260317 16:16:04.431562 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
W20260317 16:16:04.432909 17053 leader_election.cc:343] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.433571 17054 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters: dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.438560 17212 consensus_queue.cc:237] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.429562 17213 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.444265 17206 tablet_bootstrap.cc:654] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.425323 17208 tablet_bootstrap.cc:492] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.439807 17231 raft_consensus.cc:2804] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.444976 17231 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.445253 17208 ts_tablet_manager.cc:1403] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.066s user 0.015s sys 0.009s
I20260317 16:16:04.445363 17231 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.452379 17207 tablet_bootstrap.cc:492] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.452948 17207 ts_tablet_manager.cc:1403] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.081s user 0.020s sys 0.027s
I20260317 16:16:04.454449 17208 raft_consensus.cc:359] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.455649 17208 raft_consensus.cc:385] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.456452 17208 raft_consensus.cc:740] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.457393 17207 raft_consensus.cc:359] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.458613 17207 raft_consensus.cc:385] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.459053 17207 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.457505 17208 consensus_queue.cc:260] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.459223 17231 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.460028 17207 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.462729 17208 ts_tablet_manager.cc:1434] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20260317 16:16:04.465965 17208 tablet_bootstrap.cc:492] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.463080 17207 ts_tablet_manager.cc:1434] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.009s user 0.012s sys 0.000s
I20260317 16:16:04.468748 17207 tablet_bootstrap.cc:492] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.490836 17208 tablet_bootstrap.cc:654] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.491786 17207 tablet_bootstrap.cc:654] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.498879 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:04.499517 17014 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.499541 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:04.500653 17231 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
W20260317 16:16:04.501343 17053 leader_election.cc:343] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.504752 17206 tablet_bootstrap.cc:492] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.506400 17206 ts_tablet_manager.cc:1403] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.003s
I20260317 16:16:04.506839 16911 catalog_manager.cc:5671] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 1 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:04.516323 17014 raft_consensus.cc:2468] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 1.
I20260317 16:16:04.513339 17231 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.518850 17206 raft_consensus.cc:359] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.520200 17206 raft_consensus.cc:385] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.520774 17054 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters: dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.522085 17213 raft_consensus.cc:2804] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.518950 17231 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.526139 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.527096 17163 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 0.
I20260317 16:16:04.521039 17206 raft_consensus.cc:740] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.529134 17053 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:04.539378 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.539764 17206 consensus_queue.cc:260] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.539613 17231 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887)
W20260317 16:16:04.542251 17054 leader_election.cc:343] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.542645 17231 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.544059 17231 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.544620 17231 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.551913 17213 raft_consensus.cc:697] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:04.553819 17207 tablet_bootstrap.cc:492] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.553609 17213 consensus_queue.cc:237] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.555107 17207 ts_tablet_manager.cc:1403] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.007s
I20260317 16:16:04.560050 17206 ts_tablet_manager.cc:1434] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.051s user 0.008s sys 0.005s
I20260317 16:16:04.565837 17206 tablet_bootstrap.cc:492] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.566706 17208 tablet_bootstrap.cc:492] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.572450 17238 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.573282 17238 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.578649 17238 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.588268 17206 tablet_bootstrap.cc:654] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.588409 17231 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.590575 17208 ts_tablet_manager.cc:1403] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.125s user 0.026s sys 0.032s
I20260317 16:16:04.591709 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.592422 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.594748 17207 raft_consensus.cc:359] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
W20260317 16:16:04.598320 17054 leader_election.cc:343] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.598636 17207 raft_consensus.cc:385] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.599014 17207 raft_consensus.cc:740] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.599924 17208 raft_consensus.cc:359] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.601387 17208 raft_consensus.cc:385] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.601747 17208 raft_consensus.cc:740] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.605738 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:04.599754 17207 consensus_queue.cc:260] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.603550 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
W20260317 16:16:04.607100 17054 leader_election.cc:343] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.607331 17162 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.608642 17207 ts_tablet_manager.cc:1434] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.052s user 0.009s sys 0.000s
I20260317 16:16:04.611371 17214 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.612473 17207 tablet_bootstrap.cc:492] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.612212 17214 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.616101 17214 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.606015 17208 consensus_queue.cc:260] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.622160 17163 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 0.
I20260317 16:16:04.627424 17206 tablet_bootstrap.cc:492] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.628139 17206 ts_tablet_manager.cc:1403] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.020s
I20260317 16:16:04.629132 17162 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 1.
I20260317 16:16:04.622179 17208 ts_tablet_manager.cc:1434] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20260317 16:16:04.631927 17208 tablet_bootstrap.cc:492] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.633181 17206 raft_consensus.cc:359] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.634270 17206 raft_consensus.cc:385] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.604355 17231 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Requested vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887)
I20260317 16:16:04.634689 17206 raft_consensus.cc:740] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.642155 17208 tablet_bootstrap.cc:654] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.643543 17206 consensus_queue.cc:260] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.644121 17053 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.645903 17231 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.646593 17231 raft_consensus.cc:697] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:04.646915 17206 ts_tablet_manager.cc:1434] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260317 16:16:04.647919 17207 tablet_bootstrap.cc:654] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.648710 17206 tablet_bootstrap.cc:492] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.647836 17231 consensus_queue.cc:237] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.650955 17053 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.652253 17214 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.654495 17213 raft_consensus.cc:2804] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.654907 17213 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.655371 17213 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.654960 17214 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.657136 17214 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.662714 17213 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.666177 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:04.666810 17014 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.669006 17206 tablet_bootstrap.cc:654] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.678467 16910 catalog_manager.cc:5671] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 1 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:04.682085 17207 tablet_bootstrap.cc:492] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.682622 17207 ts_tablet_manager.cc:1403] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.015s
I20260317 16:16:04.683084 17214 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.683559 17214 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.685485 17207 raft_consensus.cc:359] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.686233 17207 raft_consensus.cc:385] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:16:04.686970 16959 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:16:04.687548 17214 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.674391 17014 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 1.
I20260317 16:16:04.690131 17054 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:04.686520 17207 raft_consensus.cc:740] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.691547 17238 raft_consensus.cc:2804] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.691645 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:04.692178 17238 raft_consensus.cc:697] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:04.692448 17162 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.693210 17238 consensus_queue.cc:237] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.677109 17213 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.694231 17207 consensus_queue.cc:260] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.698220 17206 tablet_bootstrap.cc:492] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.698809 17206 ts_tablet_manager.cc:1403] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.051s user 0.019s sys 0.001s
I20260317 16:16:04.699401 17207 ts_tablet_manager.cc:1434] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260317 16:16:04.700424 17207 tablet_bootstrap.cc:492] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.707924 17207 tablet_bootstrap.cc:654] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.701856 17206 raft_consensus.cc:359] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.710407 17206 raft_consensus.cc:385] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.710863 17206 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.712821 17206 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.713652 17208 tablet_bootstrap.cc:492] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.715775 17208 ts_tablet_manager.cc:1403] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.084s user 0.025s sys 0.013s
I20260317 16:16:04.718333 17206 ts_tablet_manager.cc:1434] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260317 16:16:04.719434 17208 raft_consensus.cc:359] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.722731 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.725036 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.725219 17012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.726930 17208 raft_consensus.cc:385] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.728228 17014 raft_consensus.cc:2393] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dfb44a013db548e593d221916ed0dbd7 in current term 1: Already voted for candidate c0bfe88236c24bc2971392e273683fd8 in this term.
I20260317 16:16:04.728458 17208 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.728953 17013 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 0.
W20260317 16:16:04.730676 17128 leader_election.cc:343] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.729354 17208 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.729418 17162 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 1.
I20260317 16:16:04.732376 17128 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:04.733798 17214 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.734169 17214 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.734299 17208 ts_tablet_manager.cc:1434] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.018s user 0.007s sys 0.003s
I20260317 16:16:04.734562 17214 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.736153 17206 tablet_bootstrap.cc:492] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.743161 17208 tablet_bootstrap.cc:492] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.751346 17207 tablet_bootstrap.cc:492] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:04.751979 17207 ts_tablet_manager.cc:1403] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.003s
I20260317 16:16:04.752691 17214 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.756101 17214 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.762480 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:04.762971 17252 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.763582 17014 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.763844 17252 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.769822 17207 raft_consensus.cc:359] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.773177 17207 raft_consensus.cc:385] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.775434 17238 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.775054 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.781706 17231 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.774544 17207 raft_consensus.cc:740] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.799165 17212 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.805861 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.806212 17207 consensus_queue.cc:260] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.775909 17238 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
W20260317 16:16:04.808595 17128 leader_election.cc:343] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.801578 17014 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 1.
I20260317 16:16:04.807541 17212 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.802951 17231 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.803841 17089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.809394 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:04.808660 17088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.803439 17252 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.812354 17208 tablet_bootstrap.cc:654] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.812810 17088 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 0.
I20260317 16:16:04.813036 17206 tablet_bootstrap.cc:654] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.814590 17129 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:04.815769 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.816121 17088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.810412 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.815797 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
W20260317 16:16:04.818475 17054 leader_election.cc:343] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.817039 17085 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 0.
W20260317 16:16:04.818907 16980 leader_election.cc:343] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.814328 17128 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:04.814677 17087 raft_consensus.cc:2468] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 0.
I20260317 16:16:04.814831 17212 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
W20260317 16:16:04.821542 16978 leader_election.cc:343] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.817875 17231 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.822325 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.817912 17252 raft_consensus.cc:2749] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:04.822460 16978 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.823441 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.823659 17207 ts_tablet_manager.cc:1434] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.071s user 0.010s sys 0.007s
I20260317 16:16:04.825588 17212 raft_consensus.cc:2749] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:04.824330 17214 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.823777 17238 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
W20260317 16:16:04.825829 17054 leader_election.cc:343] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.825502 17129 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.823470 17162 raft_consensus.cc:2468] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 0.
W20260317 16:16:04.830821 16978 consensus_peers.cc:597] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 -> Peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Couldn't send request to peer dfb44a013db548e593d221916ed0dbd7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:16:04.831794 17252 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.832739 17207 tablet_bootstrap.cc:492] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:04.834640 17252 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.837209 17252 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.849402 17252 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.849053 16911 catalog_manager.cc:5671] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 1 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:04.858244 17252 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.858165 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:04.856737 16911 catalog_manager.cc:5671] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 1 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
W20260317 16:16:04.860442 17128 leader_election.cc:343] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.861099 17214 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:04.862002 17214 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.868783 17208 tablet_bootstrap.cc:492] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.869297 17208 ts_tablet_manager.cc:1403] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.134s user 0.024s sys 0.019s
I20260317 16:16:04.870613 17207 tablet_bootstrap.cc:654] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.871845 17208 raft_consensus.cc:359] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.872577 17208 raft_consensus.cc:385] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.872903 17208 raft_consensus.cc:740] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
W20260317 16:16:04.873965 17129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:34563, user_credentials={real_user=slave}} blocked reactor thread for 48510.3us
I20260317 16:16:04.875000 17053 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.876065 17238 raft_consensus.cc:2804] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:04.876448 17238 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:04.876765 17238 raft_consensus.cc:3060] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.877002 17208 consensus_queue.cc:260] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.882582 17208 ts_tablet_manager.cc:1434] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.013s user 0.007s sys 0.002s
I20260317 16:16:04.883774 17208 tablet_bootstrap.cc:492] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: Bootstrap starting.
I20260317 16:16:04.883730 17238 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.886555 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
W20260317 16:16:04.887931 17054 leader_election.cc:343] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:04.891019 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:04.891688 17087 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.891841 17206 tablet_bootstrap.cc:492] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.892405 17206 ts_tablet_manager.cc:1403] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.156s user 0.017s sys 0.007s
I20260317 16:16:04.895226 17206 raft_consensus.cc:359] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.900501 17206 raft_consensus.cc:385] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.900939 17206 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.901711 17087 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 1.
I20260317 16:16:04.901734 17206 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.903029 17208 tablet_bootstrap.cc:654] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.903391 17238 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:04.904711 17206 ts_tablet_manager.cc:1434] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260317 16:16:04.904834 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.905567 17163 raft_consensus.cc:2393] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0bfe88236c24bc2971392e273683fd8 in current term 1: Already voted for candidate dfb44a013db548e593d221916ed0dbd7 in this term.
I20260317 16:16:04.905819 17206 tablet_bootstrap.cc:492] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908: Bootstrap starting.
I20260317 16:16:04.906827 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:04.907603 17163 raft_consensus.cc:3060] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.914637 17163 raft_consensus.cc:2468] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 1.
I20260317 16:16:04.915815 17129 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.916926 17255 raft_consensus.cc:2764] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260317 16:16:04.917991 17129 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.921686 17208 tablet_bootstrap.cc:492] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: No bootstrap required, opened a new log
I20260317 16:16:04.922032 17255 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.922160 17208 ts_tablet_manager.cc:1403] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.009s
I20260317 16:16:04.922566 17255 raft_consensus.cc:697] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:04.923460 17255 consensus_queue.cc:237] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.924327 17206 tablet_bootstrap.cc:654] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:04.925208 17208 raft_consensus.cc:359] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.926116 17208 raft_consensus.cc:385] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.926455 17208 raft_consensus.cc:740] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.927716 17053 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.928010 16911 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:04.929008 17238 raft_consensus.cc:2749] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:04.930114 17053 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:04.931357 17238 raft_consensus.cc:2804] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:04.931880 17238 raft_consensus.cc:697] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:04.927356 17208 consensus_queue.cc:260] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.936717 17208 ts_tablet_manager.cc:1434] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260317 16:16:04.932814 17238 consensus_queue.cc:237] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.940186 17230 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.940871 17230 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:04.943594 17214 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.944272 17214 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.945207 17230 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.946249 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:04.947559 17163 raft_consensus.cc:2393] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7365fecf4e9e48b084df0c90eff83908 in current term 1: Already voted for candidate c0bfe88236c24bc2971392e273683fd8 in this term.
I20260317 16:16:04.947926 17206 tablet_bootstrap.cc:492] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908: No bootstrap required, opened a new log
I20260317 16:16:04.948454 17206 ts_tablet_manager.cc:1403] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.008s
I20260317 16:16:04.948617 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.949443 17014 raft_consensus.cc:2393] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dfb44a013db548e593d221916ed0dbd7 in current term 1: Already voted for candidate c0bfe88236c24bc2971392e273683fd8 in this term.
I20260317 16:16:04.945938 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.951697 17206 raft_consensus.cc:359] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.952631 17206 raft_consensus.cc:385] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:04.952973 17206 raft_consensus.cc:740] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Initialized, Role: FOLLOWER
I20260317 16:16:04.953285 16980 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:04.953732 17206 consensus_queue.cc:260] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.954543 17230 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.963284 17230 raft_consensus.cc:2749] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:04.963573 17206 ts_tablet_manager.cc:1434] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260317 16:16:04.964439 17214 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:04.966101 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:04.968494 17163 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:04.976624 17212 consensus_queue.cc:1048] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:04.977224 17231 consensus_queue.cc:1048] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:04.987800 16911 catalog_manager.cc:5671] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:04.992658 17231 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:04.993472 17231 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:04.997263 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:04.998152 17014 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 0.
I20260317 16:16:04.999487 17054 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:05.000377 17213 raft_consensus.cc:2804] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:05.000730 17213 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:05.001082 17213 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.014436 17231 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.018539 17213 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.019862 17129 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:05.020925 17213 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.022330 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:05.022953 17014 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.024698 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:05.025239 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:05.028332 17207 tablet_bootstrap.cc:492] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:05.028800 17207 ts_tablet_manager.cc:1403] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.197s user 0.063s sys 0.081s
I20260317 16:16:05.031536 17207 raft_consensus.cc:359] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.032838 17207 raft_consensus.cc:385] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:05.033667 17207 raft_consensus.cc:740] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:05.034379 17207 consensus_queue.cc:260] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.042915 17207 ts_tablet_manager.cc:1434] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260317 16:16:05.046553 17252 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:05.047300 17252 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.049376 17213 consensus_queue.cc:1048] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.049963 17252 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
W20260317 16:16:05.055236 17053 leader_election.cc:343] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:05.057090 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
W20260317 16:16:05.057446 17053 leader_election.cc:343] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:05.057830 17013 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 0.
I20260317 16:16:05.059033 17128 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:05.060041 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:05.060851 17252 raft_consensus.cc:2804] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:05.061039 17085 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 0.
I20260317 16:16:05.063292 17252 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:05.065385 17252 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.073144 17014 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 1.
I20260317 16:16:05.070422 17252 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.075270 17054 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters: dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:05.076416 17238 raft_consensus.cc:2804] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:05.077260 17238 raft_consensus.cc:697] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:05.078202 17238 consensus_queue.cc:237] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.078665 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:05.079315 17085 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.082532 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:05.080093 17252 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:05.087765 17207 tablet_bootstrap.cc:492] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7: Bootstrap starting.
I20260317 16:16:05.111147 17014 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.085644 17255 raft_consensus.cc:2749] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:05.113866 17085 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 1.
I20260317 16:16:05.096998 16911 catalog_manager.cc:5671] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 1 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:05.116568 17129 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:05.105151 17230 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:05.119678 17230 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.128334 17230 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.128597 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:05.129647 17161 raft_consensus.cc:2393] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7365fecf4e9e48b084df0c90eff83908 in current term 1: Already voted for candidate c0bfe88236c24bc2971392e273683fd8 in this term.
I20260317 16:16:05.126391 17252 raft_consensus.cc:2804] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:05.133033 17252 raft_consensus.cc:697] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:05.137547 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:05.138749 17207 tablet_bootstrap.cc:654] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:05.139838 16980 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:05.142562 17230 raft_consensus.cc:3060] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.142328 17252 consensus_queue.cc:237] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.140691 17014 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 1.
I20260317 16:16:05.150177 17230 raft_consensus.cc:2749] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:05.159416 17213 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:05.159379 16908 catalog_manager.cc:5671] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:05.161162 17207 tablet_bootstrap.cc:492] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7: No bootstrap required, opened a new log
I20260317 16:16:05.161592 17207 ts_tablet_manager.cc:1403] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7: Time spent bootstrapping tablet: real 0.074s user 0.017s sys 0.008s
I20260317 16:16:05.157182 16911 catalog_manager.cc:5671] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 1 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:05.165052 17207 raft_consensus.cc:359] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.165745 17207 raft_consensus.cc:385] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:05.166025 17207 raft_consensus.cc:740] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Initialized, Role: FOLLOWER
I20260317 16:16:05.179312 17207 consensus_queue.cc:260] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.180646 17231 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.181975 17207 ts_tablet_manager.cc:1434] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7: Time spent starting tablet: real 0.020s user 0.006s sys 0.002s
I20260317 16:16:05.188683 17252 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:05.189265 17252 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.193864 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:05.193956 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:05.197731 17129 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:05.198931 17255 raft_consensus.cc:2749] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:05.199590 17252 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:05.236266 17230 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:05.236755 17230 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.238480 17230 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.239549 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:05.239621 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:05.240414 17085 raft_consensus.cc:2393] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7365fecf4e9e48b084df0c90eff83908 in current term 1: Already voted for candidate dfb44a013db548e593d221916ed0dbd7 in this term.
I20260317 16:16:05.241292 16980 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:05.241936 17230 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.249620 17230 raft_consensus.cc:2749] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:05.262033 17231 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.278036 17238 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.347983 17252 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.354174 17252 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.359552 17087 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.369863 17162 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.370615 17252 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.391983 17252 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:05.474056 17238 consensus_queue.cc:1048] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.494828 17237 consensus_queue.cc:1048] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:05.542207 17085 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94613bfab95744bca18c8c52f109387c"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:53928
I20260317 16:16:05.542924 17085 raft_consensus.cc:606] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:05.558046 17237 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:05.565220 17162 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:48880
I20260317 16:16:05.565864 17162 raft_consensus.cc:606] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:05.575621 17162 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:05.579383 17085 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:53928
I20260317 16:16:05.580056 17085 raft_consensus.cc:606] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:05.590183 17213 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:05.590895 17085 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:53928
I20260317 16:16:05.591777 17085 raft_consensus.cc:606] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:05.599520 17255 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.599690 17163 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:48880
I20260317 16:16:05.600502 17163 raft_consensus.cc:606] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:05.607755 17163 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:48880
I20260317 16:16:05.608541 17163 raft_consensus.cc:606] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:05.639020 17214 raft_consensus.cc:993] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:05.639508 17214 raft_consensus.cc:1081] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:05.647068 17087 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53926
I20260317 16:16:05.648216 17087 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:05.649065 17087 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.661113 17213 raft_consensus.cc:993] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.661772 17231 raft_consensus.cc:1081] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.665333 17214 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:05.665721 17231 raft_consensus.cc:993] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.667959 17231 raft_consensus.cc:1081] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.678921 17087 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.681653 17087 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 2 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.683420 17014 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47386
I20260317 16:16:05.683785 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:05.684044 17014 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:05.684377 17162 raft_consensus.cc:3055] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:05.684572 17014 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.684760 17162 raft_consensus.cc:740] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:05.685670 17162 consensus_queue.cc:260] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.687040 17162 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.687202 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:05.691361 17012 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94613bfab95744bca18c8c52f109387c"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47386
I20260317 16:16:05.691944 17012 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:05.692325 17012 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.694470 17162 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
I20260317 16:16:05.700101 17011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:47412
I20260317 16:16:05.700326 17053 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:05.700713 17011 raft_consensus.cc:606] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:05.701613 17231 raft_consensus.cc:2804] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:05.702198 17231 raft_consensus.cc:697] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:05.703058 17231 consensus_queue.cc:237] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.709256 17087 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:53928
I20260317 16:16:05.710110 17087 raft_consensus.cc:606] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:05.710798 17012 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:05.713533 16911 catalog_manager.cc:5671] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 1 to 2, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 2 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:05.715135 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:05.710206 17014 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.719585 17012 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Requested vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:05.719758 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:05.720644 17163 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.721306 17014 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.720508 17162 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.724452 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:05.725054 17085 raft_consensus.cc:3055] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:05.725394 17085 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:05.726182 17085 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:05.727499 17085 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.729092 17163 raft_consensus.cc:2468] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:05.729713 17162 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:05.722271 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:05.730960 17087 raft_consensus.cc:3055] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:05.731316 17087 raft_consensus.cc:740] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:05.731490 16978 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:05.732334 17230 raft_consensus.cc:2804] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:05.732173 17087 consensus_queue.cc:260] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.732625 16978 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:05.732779 17230 raft_consensus.cc:697] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:05.733564 17087 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.733801 17263 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:05.733683 17230 consensus_queue.cc:237] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.734324 17263 raft_consensus.cc:697] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:05.735321 17263 consensus_queue.cc:237] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:05.740593 17085 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:05.740850 17087 raft_consensus.cc:2468] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:05.752673 17231 raft_consensus.cc:993] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.753139 17231 raft_consensus.cc:1081] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.756104 17014 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47386
I20260317 16:16:05.756918 17014 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:05.757443 17014 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.765779 17014 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.769250 17014 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.769544 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:05.770154 17085 raft_consensus.cc:3055] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:05.770318 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:05.770617 17085 raft_consensus.cc:740] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:05.771250 17163 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.777858 17085 consensus_queue.cc:260] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.779543 17085 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.781589 17163 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:05.783879 16978 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:05.785631 17294 raft_consensus.cc:2804] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:05.786161 17294 raft_consensus.cc:697] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:05.787189 17294 consensus_queue.cc:237] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.787601 17085 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:05.796144 16911 catalog_manager.cc:5671] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 1 to 2, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 2 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:05.797750 16911 catalog_manager.cc:5671] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 1 to 2, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 2 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:05.823169 16911 catalog_manager.cc:5671] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 1 to 2, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 2 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:05.979321 17255 raft_consensus.cc:993] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:05.979801 17214 raft_consensus.cc:1081] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:05.981524 17085 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53926
I20260317 16:16:05.982067 17085 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:05.982372 17085 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.987437 17085 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:05.989163 17085 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 2 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:05.990338 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:05.990459 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:05.991494 17163 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:05.990998 17014 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:05.991827 17163 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:05.998806 17253 raft_consensus.cc:993] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.999298 17253 raft_consensus.cc:1081] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:05.999831 17014 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
I20260317 16:16:06.000913 17013 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47402
I20260317 16:16:06.001363 17054 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:05.993192 17163 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.002367 17231 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:06.002871 17163 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.001505 17013 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:06.002893 17231 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:06.003811 17013 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.004446 17231 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.010360 17163 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
I20260317 16:16:06.019127 16911 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 1 to 2, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 2 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:06.020418 17013 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.023938 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:06.024114 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:06.024636 17085 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.024772 17163 raft_consensus.cc:3055] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:06.025236 17163 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:06.026262 17163 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.027467 17163 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.033313 17013 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:06.038308 17085 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:06.039830 16980 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:06.040562 17163 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:06.041107 17263 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:06.041709 17263 raft_consensus.cc:697] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:06.042624 17263 consensus_queue.cc:237] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.050498 16911 catalog_manager.cc:5671] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 1 to 2, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 2 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:06.106379 17294 raft_consensus.cc:993] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:06.107029 17263 raft_consensus.cc:1081] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:06.108819 17085 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53920
I20260317 16:16:06.109750 17085 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:06.110468 17085 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.119912 17085 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.121917 17085 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 2 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:06.123121 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:06.123152 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:06.123736 17163 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.123917 17013 raft_consensus.cc:3055] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:06.124358 17013 raft_consensus.cc:740] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:06.125054 17013 consensus_queue.cc:260] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.126117 17013 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.131798 17163 raft_consensus.cc:2468] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
I20260317 16:16:06.134023 17053 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:06.135298 17231 raft_consensus.cc:2804] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:06.135790 17231 raft_consensus.cc:697] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:06.136435 17013 raft_consensus.cc:2468] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
I20260317 16:16:06.136562 17163 raft_consensus.cc:1275] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.136615 17231 consensus_queue.cc:237] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.138916 17263 consensus_queue.cc:1048] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.144742 17085 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.148766 17263 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.152515 17237 raft_consensus.cc:993] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:06.153542 17237 raft_consensus.cc:1081] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:06.156919 17162 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48862
I20260317 16:16:06.157584 17162 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:06.158160 17162 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.161590 17161 raft_consensus.cc:1275] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.163327 17238 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.170855 17087 raft_consensus.cc:1275] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.173285 17230 consensus_queue.cc:1048] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.178108 17160 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.180045 17230 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.203084 17013 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.216895 17013 raft_consensus.cc:1275] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.218547 17237 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.236644 17162 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.239391 17162 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 2 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:06.242939 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:06.243546 17087 raft_consensus.cc:3055] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:06.243872 17087 raft_consensus.cc:740] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:06.244638 17087 consensus_queue.cc:260] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.245909 17087 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.246848 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:06.247601 17013 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:06.253095 16909 catalog_manager.cc:5671] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 1 to 2, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 2 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:06.256273 17013 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 2.
I20260317 16:16:06.258104 17128 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:06.259094 17214 raft_consensus.cc:2804] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:06.259545 17214 raft_consensus.cc:697] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 2 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:06.260619 17214 consensus_queue.cc:237] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:06.266659 17087 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 2.
I20260317 16:16:06.269881 16909 catalog_manager.cc:5671] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 1 to 2, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:06.369881 17087 raft_consensus.cc:1275] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.371340 17230 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.375370 17161 raft_consensus.cc:1275] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.377334 17230 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.450357 17087 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.451879 17297 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:06.452785 17161 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.454113 17297 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.519588 17161 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.520817 17304 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.529161 17013 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.530882 17304 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.628039 17161 raft_consensus.cc:1275] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.629472 17304 consensus_queue.cc:1048] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:06.639129 17013 raft_consensus.cc:1275] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.640821 17013 raft_consensus.cc:1275] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.643141 17213 consensus_queue.cc:1048] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:06.643100 17214 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:06.655051 17087 raft_consensus.cc:1275] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:06.656534 17214 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:08.741813 16879 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260317 16:16:08.742497 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:08.770491 17014 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94613bfab95744bca18c8c52f109387c"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:47428
I20260317 16:16:08.771057 17014 raft_consensus.cc:606] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:08.783546 17087 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:08.784557 17087 raft_consensus.cc:606] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:08.785980 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260317 16:16:08.786576 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:08.941196 17213 raft_consensus.cc:993] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:08.941577 17213 raft_consensus.cc:1081] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:08.943336 17161 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48862
I20260317 16:16:08.943987 17161 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:08.944423 17161 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:08.950004 17161 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:08.952404 17161 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:08.953439 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:08.953310 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:08.954027 17087 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:08.954057 17014 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:08.954339 17087 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:08.955216 17087 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:08.956167 17087 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:08.960243 17014 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:08.961697 17128 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:08.963073 17322 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:08.964201 17087 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:08.965083 17322 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:08.966383 17322 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:08.978470 16909 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 2 to 3, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:09.224584 17318 raft_consensus.cc:993] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:09.225103 17318 raft_consensus.cc:1081] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:09.226994 17161 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94613bfab95744bca18c8c52f109387c"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48864
I20260317 16:16:09.227573 17161 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:09.227973 17161 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:09.232867 17161 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:09.234582 17161 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:09.235607 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:09.235597 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:09.236354 17087 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:09.236321 17013 raft_consensus.cc:3055] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:09.236893 17013 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:09.237699 17013 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:09.238945 17013 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:09.243759 17087 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:09.245115 17129 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:09.245824 17013 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:09.245996 17322 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:09.246845 17322 raft_consensus.cc:697] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:09.247946 17322 consensus_queue.cc:237] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:09.256747 16909 catalog_manager.cc:5671] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 2 to 3, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:09.444546 17087 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:09.446071 17322 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:09.460211 17013 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:09.463416 17322 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:09.690062 17087 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:09.691290 17334 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:09.701706 17013 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:09.703689 17322 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:09.787714 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:09.797427 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:09.798079 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:10.799849 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:10.809924 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:10.810649 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:11.812211 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:11.818307 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:11.819126 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:12.820924 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:12.826833 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:12.827400 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:13.828474 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:13.834838 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:13.835245 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:14.907562 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:15.293417 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:15.294057 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:15.525918 17347 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:15.527297 17347 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:15.539829 17346 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:15.540597 17346 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
W20260317 16:16:15.564396 17035 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5060 after lost signal to thread 16882
I20260317 16:16:15.611651 17346 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:15.711813 17347 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:15.926663 17347 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:15.927652 17347 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:15.937943 17347 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:15.939569 17346 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:15.970151 17346 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:15.972865 17346 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.030731 17349 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:16.031538 17349 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:16.036327 17349 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887)
I20260317 16:16:16.037045 17349 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:16.037612 17349 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.042055 17349 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.067735 17349 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:16.069988 17349 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.072084 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.072010 17088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.073089 17086 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 3.
I20260317 16:16:16.077230 17088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.077811 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.074883 17349 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.077850 16980 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:16.078696 17086 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 3.
I20260317 16:16:16.077891 17088 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 2.
I20260317 16:16:16.080171 17344 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:16:16.080518 17344 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:16.080322 16980 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:16.080871 17344 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:16.081447 17347 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:16:16.081911 17347 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:16.082068 16980 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:16.082391 17347 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:16.083128 17346 raft_consensus.cc:2804] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:16:16.083496 17346 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:16.083884 17346 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.091076 17344 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.093147 17346 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.097911 17349 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:16.101025 17349 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.098701 17344 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.107427 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:16.108707 17349 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.108970 17344 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:16.118538 17344 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.121717 17344 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.124584 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.128521 17346 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.130831 17344 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:16.131459 17088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:16.131421 17344 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.132362 17088 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.141399 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.135674 17086 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:16.154982 17012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.158349 17089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.146593 17088 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 3.
I20260317 16:16:16.160017 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.160933 17014 raft_consensus.cc:2393] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0bfe88236c24bc2971392e273683fd8 in current term 3: Already voted for candidate 7365fecf4e9e48b084df0c90eff83908 in this term.
I20260317 16:16:16.161329 16980 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:16.161650 17010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.167256 17322 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:16.167879 17322 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.170855 17322 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:16.171445 17322 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:16.172092 17322 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.179632 17344 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.183557 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.184705 17322 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:16.185328 17086 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 4.
I20260317 16:16:16.185524 17353 raft_consensus.cc:2804] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:16.187942 17353 raft_consensus.cc:697] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:16.189101 17353 consensus_queue.cc:237] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.191303 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.192021 17347 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:16.193379 17013 raft_consensus.cc:2393] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0bfe88236c24bc2971392e273683fd8 in current term 4: Already voted for candidate 7365fecf4e9e48b084df0c90eff83908 in this term.
I20260317 16:16:16.196264 17356 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:16.196774 17356 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
W20260317 16:16:16.197712 17035 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5740 after lost signal to thread 16945
W20260317 16:16:16.198874 17035 debug-util.cc:398] Leaking SignalData structure 0x7b0800109580 after lost signal to thread 17038
I20260317 16:16:16.199010 17356 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:16.200407 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.201000 17161 raft_consensus.cc:2468] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
W20260317 16:16:16.201292 17035 debug-util.cc:398] Leaking SignalData structure 0x7b08000a3de0 after lost signal to thread 17113
W20260317 16:16:16.202207 17035 debug-util.cc:398] Leaking SignalData structure 0x7b080019b700 after lost signal to thread 17187
I20260317 16:16:16.202913 17010 raft_consensus.cc:1240] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Rejecting Update request from peer dfb44a013db548e593d221916ed0dbd7 for earlier term 3. Current term is 4. Ops: []
I20260317 16:16:16.203009 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.204288 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.204561 16980 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:16.204897 17161 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 2.
I20260317 16:16:16.205778 17355 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:16.205770 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.209381 17089 raft_consensus.cc:1240] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Rejecting Update request from peer dfb44a013db548e593d221916ed0dbd7 for earlier term 3. Current term is 4. Ops: []
I20260317 16:16:16.207280 17053 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:16.206907 17355 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:16.211244 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.211424 17356 consensus_queue.cc:1059] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:16.211835 17349 raft_consensus.cc:2749] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.211593 17355 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.210649 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.211176 17053 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:16.214066 17354 raft_consensus.cc:3055] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:16.214429 17354 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:16.214462 17351 raft_consensus.cc:2804] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:16:16.214990 17351 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:16.215415 17351 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.215814 17088 raft_consensus.cc:1240] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Rejecting Update request from peer dfb44a013db548e593d221916ed0dbd7 for earlier term 2. Current term is 3. Ops: []
I20260317 16:16:16.215474 17354 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:16.216868 17354 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:16.219151 17053 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:16.220973 17350 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:16:16.221374 17350 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:16.221755 17350 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.209235 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.223598 17357 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:16.224820 17357 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.235721 17054 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:16.238199 17349 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:16.233304 17350 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.237298 17347 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 election: Requested vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:16.233573 17351 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.250522 17357 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:16.246806 17356 consensus_queue.cc:1059] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260317 16:16:16.252084 17357 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:16.252328 17351 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.252439 17013 raft_consensus.cc:1240] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Rejecting Update request from peer dfb44a013db548e593d221916ed0dbd7 for earlier term 3. Current term is 4. Ops: []
I20260317 16:16:16.252444 17357 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:16.238389 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.253927 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:16.253921 17357 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.258499 17129 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:16.265983 17350 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:16.271456 17356 raft_consensus.cc:2749] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:16:16.278399 17011 raft_consensus.cc:1240] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Rejecting Update request from peer dfb44a013db548e593d221916ed0dbd7 for earlier term 2. Current term is 3. Ops: []
I20260317 16:16:16.274607 17356 consensus_queue.cc:1059] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260317 16:16:16.279721 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:16.280004 17356 raft_consensus.cc:3055] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:16.280355 17356 raft_consensus.cc:740] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:16.278126 17129 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:16.281119 17356 consensus_queue.cc:260] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.282346 17356 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.286633 17161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.283691 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:16.286252 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.273211 17357 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:16.285791 17160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.282557 17129 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:16.295182 17158 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.295518 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:16.285663 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.300671 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:16.301195 17163 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 4.
I20260317 16:16:16.282704 17322 raft_consensus.cc:2749] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.302556 17322 raft_consensus.cc:2749] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.274134 17349 raft_consensus.cc:2749] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:16:16.303432 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:16.303870 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:16.290542 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:16.309053 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:16.311383 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:16.312072 17163 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.311724 16911 catalog_manager.cc:5671] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 2 to 3, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 3 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:16.316618 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:16.316784 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:16.317634 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:16.320595 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:16.316850 17086 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 4.
W20260317 16:16:16.264092 17354 consensus_queue.cc:1175] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7365fecf4e9e48b084df0c90eff83908" responder_term: 4 status { last_received { term: 4 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer dfb44a013db548e593d221916ed0dbd7 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260317 16:16:16.317097 17129 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:16.324834 17322 raft_consensus.cc:2749] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.327113 16978 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:16.328197 17353 raft_consensus.cc:2749] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.329052 16978 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:16.330101 17353 raft_consensus.cc:2749] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.335224 17053 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:16.336138 17349 raft_consensus.cc:2749] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:16:16.336934 17163 raft_consensus.cc:2468] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 3.
I20260317 16:16:16.338166 17160 raft_consensus.cc:1240] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Rejecting Update request from peer 7365fecf4e9e48b084df0c90eff83908 for earlier term 2. Current term is 3. Ops: []
I20260317 16:16:16.338598 17161 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 4.
I20260317 16:16:16.339913 17162 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 3.
I20260317 16:16:16.343128 16978 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:16.344012 17353 raft_consensus.cc:2749] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:16:16.345439 17353 consensus_queue.cc:1059] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:16.346446 16980 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:16.346902 17353 raft_consensus.cc:3055] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:16.347226 17159 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 4.
I20260317 16:16:16.347281 17353 raft_consensus.cc:740] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:16.347374 17347 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:16.348104 17347 raft_consensus.cc:697] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:16.348289 17353 consensus_queue.cc:260] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.349395 17053 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:16.349136 17347 consensus_queue.cc:237] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:16.350680 17349 raft_consensus.cc:2804] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:16.351575 17349 raft_consensus.cc:697] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:16.354151 17353 raft_consensus.cc:3060] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.353197 17349 consensus_queue.cc:237] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.371261 16909 catalog_manager.cc:5671] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 2 to 3, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 3 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:16.386289 16909 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 3 to 4, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 4 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:16.388291 16909 catalog_manager.cc:5671] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 3 to 4, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 4 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:16.599438 17086 raft_consensus.cc:1275] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:16.600796 17353 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:16.623206 17159 raft_consensus.cc:1275] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:16.624886 17355 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:16.638980 17086 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:16.640462 17355 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:16.653159 17159 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:16.658026 17347 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:16.663494 17087 raft_consensus.cc:1240] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Rejecting Update request from peer 7365fecf4e9e48b084df0c90eff83908 for earlier term 2. Current term is 3. Ops: []
I20260317 16:16:16.666702 17352 consensus_queue.cc:1059] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:16.668072 17347 raft_consensus.cc:3055] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:16.668632 17347 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:16.669540 17347 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:16.671440 17347 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:16.862375 17011 raft_consensus.cc:1275] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:16.863682 17360 consensus_queue.cc:1048] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:16.865260 17163 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:16.867072 17347 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:16.877439 17159 raft_consensus.cc:1275] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:16.883111 17360 consensus_queue.cc:1048] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:16.906776 17086 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:16.909818 17353 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:17.319175 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:17.329747 17086 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:17.330348 17086 raft_consensus.cc:606] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:17.332983 17011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:47428
I20260317 16:16:17.333484 17011 raft_consensus.cc:606] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:17.335819 17011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:47428
I20260317 16:16:17.336304 17011 raft_consensus.cc:606] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:17.337354 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260317 16:16:17.337765 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:17.573272 17353 raft_consensus.cc:993] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:17.574196 17352 raft_consensus.cc:1081] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:17.576582 17159 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48864
I20260317 16:16:17.577144 17159 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:17.577481 17159 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:17.583063 17159 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.584885 17159 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 4 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:17.585796 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:17.586216 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:17.586580 17011 raft_consensus.cc:3055] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:17.586786 17086 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:17.586993 17011 raft_consensus.cc:740] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:17.588359 17011 consensus_queue.cc:260] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.589681 17011 raft_consensus.cc:3060] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:17.596374 17086 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 4.
I20260317 16:16:17.598701 17011 raft_consensus.cc:2468] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 4.
I20260317 16:16:17.599215 17129 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:17.603505 17381 raft_consensus.cc:2804] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:17.604616 17381 raft_consensus.cc:697] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 4 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:17.605648 17381 consensus_queue.cc:237] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.614434 16908 catalog_manager.cc:5671] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 3 to 4, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 4 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:17.661368 17360 raft_consensus.cc:993] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:17.661864 17360 raft_consensus.cc:1081] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:17.666371 17159 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48862
I20260317 16:16:17.668303 17159 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:17.669088 17159 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:17.676469 17159 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.678146 17159 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:17.679608 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:17.680205 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:17.680511 17011 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:17.680927 17086 raft_consensus.cc:3055] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:17.681654 17086 raft_consensus.cc:740] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:17.682541 17086 consensus_queue.cc:260] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.684687 17086 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:17.688205 17011 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:17.697531 17128 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:17.698655 17381 raft_consensus.cc:2804] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:17.699199 17381 raft_consensus.cc:697] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:17.700147 17381 consensus_queue.cc:237] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.701761 17086 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:17.711377 16908 catalog_manager.cc:5671] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 2 to 3, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:17.770536 17353 raft_consensus.cc:993] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:17.771219 17352 raft_consensus.cc:1081] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:17.773813 17159 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48864
I20260317 16:16:17.775198 17159 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:17.775755 17159 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:17.784627 17159 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.785475 17381 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:17.785974 17381 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.787442 17159 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 5 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:17.788569 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:17.789201 17011 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:16:17.788589 17381 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:17.789892 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:17.790104 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:17.790457 17086 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:17.789535 17011 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:17.791356 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:17.792037 17087 raft_consensus.cc:2393] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dfb44a013db548e593d221916ed0dbd7 in current term 3: Already voted for candidate c0bfe88236c24bc2971392e273683fd8 in this term.
I20260317 16:16:17.791651 17011 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.793550 17011 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:17.790693 17014 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:17.795804 17128 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters: c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:17.797081 17381 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:16:17.797550 17381 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:17.797907 17381 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:17.799441 17086 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 5.
I20260317 16:16:17.800812 17129 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:17.801673 17382 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:16:17.801597 17011 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 5.
I20260317 16:16:17.804389 17382 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:17.805457 17382 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.807075 17381 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.811836 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:17.811861 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:17.813196 17086 raft_consensus.cc:2393] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dfb44a013db548e593d221916ed0dbd7 in current term 3: Already voted for candidate c0bfe88236c24bc2971392e273683fd8 in this term.
I20260317 16:16:17.811946 17381 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:17.816057 16908 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 4 to 5, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 5 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:17.819602 17011 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 3.
I20260317 16:16:17.820695 17128 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters: c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:17.821429 17381 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:17.822275 17381 raft_consensus.cc:697] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:17.823235 17381 consensus_queue.cc:237] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:17.831321 16908 catalog_manager.cc:5671] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 2 to 3, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:18.059223 17011 raft_consensus.cc:1275] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:18.060673 17381 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:18.074051 17086 raft_consensus.cc:1275] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:18.075775 17382 consensus_queue.cc:1048] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:18.118043 17011 raft_consensus.cc:1275] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:18.119339 17382 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:18.131624 17086 raft_consensus.cc:1275] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:18.134292 17382 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:18.256824 17011 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260317 16:16:18.258095 17382 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:16:18.271719 17086 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260317 16:16:18.275367 17381 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:16:18.287614 17086 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:18.288955 17381 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:18.300619 17011 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:18.302248 17388 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:18.338721 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:18.359285 17159 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:44740
I20260317 16:16:18.359813 17159 raft_consensus.cc:606] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:18.361790 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260317 16:16:18.362275 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:18.795279 17388 raft_consensus.cc:993] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:18.795822 17382 raft_consensus.cc:1081] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:18.798357 17011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47402
I20260317 16:16:18.799366 17011 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:18.799793 17011 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:16:18.808844 17011 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:18.811610 17011 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 6 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:18.812652 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:18.812765 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:18.813342 17086 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:16:18.813385 17159 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:16:18.813884 17159 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:18.814736 17159 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:18.816277 17159 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:16:18.820972 17086 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 6.
I20260317 16:16:18.822530 16980 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:18.823400 17352 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 FOLLOWER]: Leader election won for term 6
I20260317 16:16:18.823475 17159 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 6.
I20260317 16:16:18.824787 17352 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:18.825939 17352 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:18.833619 16909 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 5 to 6, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 6 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:19.277491 17086 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 6 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260317 16:16:19.279384 17352 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:16:19.293040 17159 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 6 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260317 16:16:19.294308 17352 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:16:19.391314 16879 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260317 16:16:19.396222 17014 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94613bfab95744bca18c8c52f109387c"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:47412
I20260317 16:16:19.396934 17014 raft_consensus.cc:606] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:19.401968 17159 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:48880
I20260317 16:16:19.402606 17159 raft_consensus.cc:606] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:19.407936 17014 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:47412
I20260317 16:16:19.408432 17014 raft_consensus.cc:606] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:19.415884 17014 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:47412
I20260317 16:16:19.416374 17014 raft_consensus.cc:606] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:19.421075 17159 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
mode: GRACEFUL
new_leader_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:48880
I20260317 16:16:19.421654 17159 raft_consensus.cc:606] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Received request to transfer leadership to TS c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:19.442315 17405 raft_consensus.cc:993] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.442797 17352 raft_consensus.cc:1081] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.444386 17087 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94613bfab95744bca18c8c52f109387c"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53920
I20260317 16:16:19.444761 17087 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:19.444972 17087 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:19.449225 17087 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:19.451150 17087 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 5 election: Requested vote from peers dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361), 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887)
I20260317 16:16:19.452040 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:19.452244 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:19.452581 17159 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:19.452780 17014 raft_consensus.cc:3055] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:16:19.453104 17014 raft_consensus.cc:740] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:19.453720 17014 consensus_queue.cc:260] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:19.454824 17014 raft_consensus.cc:3060] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:19.458207 17159 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 5.
I20260317 16:16:19.459709 17053 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:19.460660 17360 raft_consensus.cc:2804] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:16:19.461352 17014 raft_consensus.cc:2468] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 5.
I20260317 16:16:19.461184 17360 raft_consensus.cc:697] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 5 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:19.462713 17360 consensus_queue.cc:237] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:19.472239 16908 catalog_manager.cc:5671] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 4 to 5, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 5 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:19.507277 17388 raft_consensus.cc:993] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.507746 17388 raft_consensus.cc:1081] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.509282 17087 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53926
I20260317 16:16:19.509774 17087 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:19.510128 17087 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:19.516424 17087 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.517881 17087 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 4 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:19.519001 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:19.519023 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:19.519516 17014 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:19.519738 17159 raft_consensus.cc:3055] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:19.520144 17159 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:19.520793 17159 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.521788 17159 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:19.527076 17159 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 4.
I20260317 16:16:19.527778 17014 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 4.
I20260317 16:16:19.529155 17053 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:19.530026 17360 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:19.530484 17360 raft_consensus.cc:697] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:19.531575 17360 consensus_queue.cc:237] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.539278 16908 catalog_manager.cc:5671] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 3 to 4, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 4 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:19.679738 17382 raft_consensus.cc:993] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.680269 17388 raft_consensus.cc:1081] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.682466 17087 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53926
I20260317 16:16:19.683121 17087 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:19.683710 17087 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:19.689350 17087 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.691161 17087 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 4 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:19.692137 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:19.692242 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:19.692701 17159 raft_consensus.cc:3055] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:19.692840 17014 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:19.693017 17159 raft_consensus.cc:740] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:19.693861 17159 consensus_queue.cc:260] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.694881 17159 raft_consensus.cc:3060] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:19.697713 17014 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 4.
I20260317 16:16:19.698689 17054 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8; no voters:
I20260317 16:16:19.699366 17360 raft_consensus.cc:2804] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:19.700053 17360 raft_consensus.cc:697] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:19.700301 17159 raft_consensus.cc:2468] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 4.
I20260317 16:16:19.700917 17360 consensus_queue.cc:237] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.708084 16908 catalog_manager.cc:5671] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 3 to 4, leader changed from dfb44a013db548e593d221916ed0dbd7 (127.16.123.195) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 4 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:19.783352 17405 raft_consensus.cc:993] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.783802 17352 raft_consensus.cc:1081] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.785302 17087 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53920
I20260317 16:16:19.786047 17087 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:19.786371 17087 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:19.788465 17405 raft_consensus.cc:993] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: : Instructing follower c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.788983 17352 raft_consensus.cc:1081] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 LEADER]: Signalling peer c0bfe88236c24bc2971392e273683fd8 to start an election
I20260317 16:16:19.790570 17086 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
from {username='slave'} at 127.0.0.1:53920
I20260317 16:16:19.791157 17086 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:19.791496 17086 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 6 FOLLOWER]: Advancing to term 7
I20260317 16:16:19.792480 17087 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.794736 17087 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:19.795125 17014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:19.795346 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:19.795715 17014 raft_consensus.cc:3055] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:19.796128 17014 raft_consensus.cc:740] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:19.796192 17159 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:19.796962 17014 consensus_queue.cc:260] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.797837 17086 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.798226 17014 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:19.799866 17086 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 7 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:19.800426 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:19.800665 17163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:19.801015 17011 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 LEADER]: Stepping down as leader of term 6
I20260317 16:16:19.801195 17163 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 6 FOLLOWER]: Advancing to term 7
I20260317 16:16:19.801389 17011 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:19.802342 17011 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.804147 17011 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 6 FOLLOWER]: Advancing to term 7
I20260317 16:16:19.804318 17159 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 3.
I20260317 16:16:19.805667 17053 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:19.806643 17360 raft_consensus.cc:2804] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:19.806737 17014 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 3.
I20260317 16:16:19.807178 17360 raft_consensus.cc:697] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:19.808058 17360 consensus_queue.cc:237] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.808681 17163 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 7.
I20260317 16:16:19.809724 17053 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:19.810439 17411 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 FOLLOWER]: Leader election won for term 7
I20260317 16:16:19.811584 17411 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 LEADER]: Becoming Leader. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:19.812386 17411 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:19.813309 17011 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0bfe88236c24bc2971392e273683fd8 in term 7.
I20260317 16:16:19.817615 16908 catalog_manager.cc:5671] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 2 to 3, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 3 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:19.829988 16908 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 reported cstate change: term changed from 6 to 7, leader changed from 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193) to c0bfe88236c24bc2971392e273683fd8 (127.16.123.194). New cstate: current_term: 7 leader_uuid: "c0bfe88236c24bc2971392e273683fd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:19.872218 17011 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260317 16:16:19.873859 17360 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:16:19.885164 17159 raft_consensus.cc:1275] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260317 16:16:19.887131 17411 consensus_queue.cc:1048] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260317 16:16:20.014173 17011 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:20.015973 17411 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:20.031180 17159 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:20.033178 17420 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:20.227900 17011 raft_consensus.cc:1275] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:20.229614 17420 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:16:20.241971 17159 raft_consensus.cc:1275] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:20.244881 17411 consensus_queue.cc:1048] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:16:20.400940 17159 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 7 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260317 16:16:20.403137 17428 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260317 16:16:20.403635 17011 raft_consensus.cc:1275] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:20.405910 17428 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:20.412248 17014 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 7 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260317 16:16:20.418607 17420 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260317 16:16:20.431602 17163 raft_consensus.cc:1275] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Refusing update from remote peer c0bfe88236c24bc2971392e273683fd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:20.449563 17360 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260317 16:16:22.469007 16879 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260317 16:16:22.470253 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:22.479537 17086 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:22.480350 17086 raft_consensus.cc:606] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:22.483325 17086 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:22.483887 17086 raft_consensus.cc:606] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:22.487160 17086 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:22.487867 17086 raft_consensus.cc:606] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:22.490626 17086 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:22.491361 17086 raft_consensus.cc:606] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Received request to transfer leadership to TS dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:22.494462 17086 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10"
dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
mode: GRACEFUL
new_leader_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:53938
I20260317 16:16:22.495286 17086 raft_consensus.cc:606] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Received request to transfer leadership to TS 7365fecf4e9e48b084df0c90eff83908
I20260317 16:16:22.496587 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260317 16:16:22.497155 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:22.531719 17439 raft_consensus.cc:993] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:22.532286 17434 raft_consensus.cc:1081] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:22.533778 17159 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48862
I20260317 16:16:22.534193 17159 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:22.534461 17159 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:22.540488 17159 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.542440 17159 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 4 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:22.543948 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:22.543855 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:22.544983 17011 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:22.545109 17086 raft_consensus.cc:3055] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:22.545819 17086 raft_consensus.cc:740] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:22.546721 17086 consensus_queue.cc:260] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.548434 17086 raft_consensus.cc:3060] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:22.551687 17011 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 4.
I20260317 16:16:22.553370 17128 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:22.554443 17388 raft_consensus.cc:2804] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:22.554311 17086 raft_consensus.cc:2468] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 4.
I20260317 16:16:22.555909 17388 raft_consensus.cc:697] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 4 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:22.557047 17388 consensus_queue.cc:237] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.565225 16909 catalog_manager.cc:5671] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 3 to 4, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 4 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:22.579026 17439 raft_consensus.cc:993] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:22.579694 17434 raft_consensus.cc:1081] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:22.582119 17011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47386
I20260317 16:16:22.582574 17011 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:22.582902 17011 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 7 FOLLOWER]: Advancing to term 8
I20260317 16:16:22.587707 17011 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.589699 17011 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 8 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:22.591042 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:22.591815 17086 raft_consensus.cc:3055] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 LEADER]: Stepping down as leader of term 7
I20260317 16:16:22.592123 17086 raft_consensus.cc:740] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:22.592020 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:22.593250 17159 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 7 FOLLOWER]: Advancing to term 8
I20260317 16:16:22.593554 17086 consensus_queue.cc:260] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.594983 17086 raft_consensus.cc:3060] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 7 FOLLOWER]: Advancing to term 8
I20260317 16:16:22.600871 17159 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 8.
I20260317 16:16:22.602391 17086 raft_consensus.cc:2468] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 8.
I20260317 16:16:22.602878 16978 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:22.605085 17446 raft_consensus.cc:2804] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 8 FOLLOWER]: Leader election won for term 8
I20260317 16:16:22.606644 17446 raft_consensus.cc:697] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 8 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:22.607795 17446 consensus_queue.cc:237] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.616474 16909 catalog_manager.cc:5671] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 7 to 8, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 8 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:22.679895 17439 raft_consensus.cc:993] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:22.680438 17439 raft_consensus.cc:1081] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:22.682271 17011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47386
I20260317 16:16:22.682996 17011 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:22.683257 17011 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:22.688707 17011 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.690371 17011 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 5 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:22.691593 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:22.691778 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:22.692122 17159 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:22.692359 17086 raft_consensus.cc:3055] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:16:22.692627 17086 raft_consensus.cc:740] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:22.693226 17086 consensus_queue.cc:260] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.694370 17086 raft_consensus.cc:3060] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:16:22.698614 17159 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 5.
I20260317 16:16:22.699153 17086 raft_consensus.cc:2468] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 5.
I20260317 16:16:22.700280 16978 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:22.701089 17446 raft_consensus.cc:2804] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:16:22.701624 17446 raft_consensus.cc:697] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 5 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:22.702454 17446 consensus_queue.cc:237] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.710459 16909 catalog_manager.cc:5671] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 4 to 5, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 5 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:22.808558 17434 raft_consensus.cc:993] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: : Instructing follower 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:22.809134 17434 raft_consensus.cc:1081] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Signalling peer 7365fecf4e9e48b084df0c90eff83908 to start an election
I20260317 16:16:22.810809 17011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10"
dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
from {username='slave'} at 127.0.0.1:47386
I20260317 16:16:22.811362 17011 raft_consensus.cc:493] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:22.811686 17011 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:22.817361 17011 raft_consensus.cc:515] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.819149 17011 leader_election.cc:290] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 election: Requested vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:22.820380 17159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
I20260317 16:16:22.820380 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aff0d35de1f84b6d9e26764809b83e10" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:22.821120 17159 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:22.821139 17086 raft_consensus.cc:3055] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:22.821734 17086 raft_consensus.cc:740] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:22.822494 17086 consensus_queue.cc:260] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.824023 17086 raft_consensus.cc:3060] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:22.830307 17159 raft_consensus.cc:2468] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 3.
I20260317 16:16:22.831854 16978 leader_election.cc:304] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:22.832088 17086 raft_consensus.cc:2468] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365fecf4e9e48b084df0c90eff83908 in term 3.
I20260317 16:16:22.832759 17446 raft_consensus.cc:2804] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:22.833279 17446 raft_consensus.cc:697] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Becoming Leader. State: Replica: 7365fecf4e9e48b084df0c90eff83908, State: Running, Role: LEADER
I20260317 16:16:22.834182 17446 consensus_queue.cc:237] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.842938 16909 catalog_manager.cc:5671] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 reported cstate change: term changed from 2 to 3, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193). New cstate: current_term: 3 leader_uuid: "7365fecf4e9e48b084df0c90eff83908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:22.844647 17439 raft_consensus.cc:993] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: : Instructing follower dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:22.845391 17434 raft_consensus.cc:1081] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Signalling peer dfb44a013db548e593d221916ed0dbd7 to start an election
I20260317 16:16:22.847357 17159 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e"
dest_uuid: "dfb44a013db548e593d221916ed0dbd7"
from {username='slave'} at 127.0.0.1:48862
I20260317 16:16:22.847976 17159 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:22.848351 17159 raft_consensus.cc:3060] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:22.854697 17159 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.856348 17159 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 2 election: Requested vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:22.857425 17011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7365fecf4e9e48b084df0c90eff83908"
I20260317 16:16:22.857425 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0bfe88236c24bc2971392e273683fd8"
I20260317 16:16:22.858240 17086 raft_consensus.cc:3055] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:22.858240 17011 raft_consensus.cc:3060] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:22.858701 17086 raft_consensus.cc:740] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0bfe88236c24bc2971392e273683fd8, State: Running, Role: LEADER
I20260317 16:16:22.859478 17086 consensus_queue.cc:260] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.860671 17086 raft_consensus.cc:3060] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:22.865823 17011 raft_consensus.cc:2468] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 2.
I20260317 16:16:22.867262 17128 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7; no voters:
I20260317 16:16:22.867645 17086 raft_consensus.cc:2468] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb44a013db548e593d221916ed0dbd7 in term 2.
I20260317 16:16:22.868399 17388 raft_consensus.cc:2804] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:22.868964 17388 raft_consensus.cc:697] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 2 LEADER]: Becoming Leader. State: Replica: dfb44a013db548e593d221916ed0dbd7, State: Running, Role: LEADER
I20260317 16:16:22.869891 17388 consensus_queue.cc:237] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:22.878109 16909 catalog_manager.cc:5671] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 reported cstate change: term changed from 1 to 2, leader changed from c0bfe88236c24bc2971392e273683fd8 (127.16.123.194) to dfb44a013db548e593d221916ed0dbd7 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "dfb44a013db548e593d221916ed0dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:23.005580 17011 raft_consensus.cc:1275] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:23.007009 17388 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:23.018383 17086 raft_consensus.cc:1275] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:23.020917 17454 consensus_queue.cc:1048] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:16:23.161520 17159 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 8 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260317 16:16:23.163239 17446 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260317 16:16:23.175967 17086 raft_consensus.cc:1275] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260317 16:16:23.177248 17446 consensus_queue.cc:1048] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260317 16:16:23.188341 17159 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260317 16:16:23.190153 17447 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:16:23.201850 17086 raft_consensus.cc:1275] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260317 16:16:23.203286 17459 consensus_queue.cc:1048] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:16:23.263554 17011 raft_consensus.cc:1275] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:23.264783 17454 consensus_queue.cc:1048] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:23.274047 17086 raft_consensus.cc:1275] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Refusing update from remote peer dfb44a013db548e593d221916ed0dbd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:23.275434 17454 consensus_queue.cc:1048] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:23.327898 17086 raft_consensus.cc:1275] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:23.329226 17459 consensus_queue.cc:1048] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:23.338507 17159 raft_consensus.cc:1275] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Refusing update from remote peer 7365fecf4e9e48b084df0c90eff83908: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:23.339999 17447 consensus_queue.cc:1048] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:23.498546 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:23.504621 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:23.505231 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:24.506682 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:24.514003 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:24.515120 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:25.516438 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:25.523375 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:25.523906 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:26.525269 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:26.531095 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:26.531526 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:27.532703 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:27.537986 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:27.538527 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:28.540057 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:28.549222 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:28.549715 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:29.551033 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:29.557541 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:29.558022 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:30.559648 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:30.565050 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:30.565830 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:31.567584 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:31.573562 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:31.574050 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:33.298538 17477 raft_consensus.cc:493] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:33.299525 17477 raft_consensus.cc:515] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } } peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } }
I20260317 16:16:33.309429 17477 leader_election.cc:290] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:33.335403 17478 raft_consensus.cc:493] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:33.335997 17478 raft_consensus.cc:515] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.338434 17478 leader_election.cc:290] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:33.370344 17477 raft_consensus.cc:493] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:33.371043 17477 raft_consensus.cc:515] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.373385 17477 leader_election.cc:290] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563)
I20260317 16:16:33.375164 17088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
W20260317 16:16:33.378940 17035 debug-util.cc:398] Leaking SignalData structure 0x7b080017aae0 after lost signal to thread 16882
I20260317 16:16:33.379187 17085 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
W20260317 16:16:33.380174 17035 debug-util.cc:398] Leaking SignalData structure 0x7b08001bb8e0 after lost signal to thread 16945
W20260317 16:16:33.381315 17035 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5180 after lost signal to thread 17038
W20260317 16:16:33.382213 17035 debug-util.cc:398] Leaking SignalData structure 0x7b0800108700 after lost signal to thread 17113
I20260317 16:16:33.386432 17478 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:33.387004 17478 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.389580 17478 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:33.391104 17160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:33.391139 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:33.394985 17481 raft_consensus.cc:493] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:33.395620 17481 raft_consensus.cc:515] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.396564 17010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9527b12d0cba460ea05b4ddcf46a0568" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:33.397235 17012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:33.397846 17481 leader_election.cc:290] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:33.404809 17054 leader_election.cc:304] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
W20260317 16:16:33.406997 17035 debug-util.cc:398] Leaking SignalData structure 0x7b08001c9040 after lost signal to thread 17187
I20260317 16:16:33.419023 17478 raft_consensus.cc:2749] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260317 16:16:33.406293 17054 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:33.421782 17478 raft_consensus.cc:2749] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:16:33.425915 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:33.426445 17469 raft_consensus.cc:493] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:33.426344 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16a83877f56c4ad7be4fdfb90a68005e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:33.427060 17469 raft_consensus.cc:515] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.429207 17469 leader_election.cc:290] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:33.429661 17469 raft_consensus.cc:493] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0bfe88236c24bc2971392e273683fd8)
I20260317 16:16:33.434223 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:33.430244 17469 raft_consensus.cc:515] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.442278 17469 leader_election.cc:290] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:33.442901 17469 raft_consensus.cc:493] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dfb44a013db548e593d221916ed0dbd7)
I20260317 16:16:33.443511 17469 raft_consensus.cc:515] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:33.451445 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:33.452405 17086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cde56c3fe80f4f94a1981c90ca3eb089" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:33.454013 17087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0bfe88236c24bc2971392e273683fd8" is_pre_election: true
I20260317 16:16:33.454484 17469 leader_election.cc:290] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:33.461735 16978 leader_election.cc:304] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:33.463690 16978 leader_election.cc:304] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:33.465953 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae772e8721354fe2bffffe3b986eaf8e" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:33.466490 17486 raft_consensus.cc:2749] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:16:33.468199 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94613bfab95744bca18c8c52f109387c" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:33.468623 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "995dacb7e66f4c47842e9eba5645d8c6" candidate_uuid: "7365fecf4e9e48b084df0c90eff83908" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
I20260317 16:16:33.470093 16978 leader_election.cc:304] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:33.470294 17013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e2510c1001e4134bf6f61fa639c3551" candidate_uuid: "dfb44a013db548e593d221916ed0dbd7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "7365fecf4e9e48b084df0c90eff83908" is_pre_election: true
I20260317 16:16:33.471417 16978 leader_election.cc:304] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7365fecf4e9e48b084df0c90eff83908; no voters: c0bfe88236c24bc2971392e273683fd8, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:33.472185 17482 raft_consensus.cc:2749] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:16:33.472860 17128 leader_election.cc:304] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:33.473176 17482 raft_consensus.cc:2749] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:16:33.473593 17485 raft_consensus.cc:2749] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:16:33.473969 17483 raft_consensus.cc:2749] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:16:33.474175 17128 leader_election.cc:304] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfb44a013db548e593d221916ed0dbd7; no voters: 7365fecf4e9e48b084df0c90eff83908, c0bfe88236c24bc2971392e273683fd8
I20260317 16:16:33.474967 17483 raft_consensus.cc:2749] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:16:33.498054 16879 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260317 16:16:33.498936 16879 tablet_server.cc:178] TabletServer@127.16.123.193:0 shutting down...
I20260317 16:16:33.542616 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:16:33.543731 16879 tablet_replica.cc:333] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.544997 16879 raft_consensus.cc:2243] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:33.545823 16879 raft_consensus.cc:2272] T 94613bfab95744bca18c8c52f109387c P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.549547 16879 tablet_replica.cc:333] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.550406 16879 raft_consensus.cc:2243] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:33.551165 16879 raft_consensus.cc:2272] T cde56c3fe80f4f94a1981c90ca3eb089 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.553802 16879 tablet_replica.cc:333] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.554574 16879 raft_consensus.cc:2243] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 5 LEADER]: Raft consensus shutting down.
I20260317 16:16:33.555804 16879 raft_consensus.cc:2272] T 7d52fb95165d493db57d26439ab3b47d P 7365fecf4e9e48b084df0c90eff83908 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.558414 16879 tablet_replica.cc:333] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.559144 16879 raft_consensus.cc:2243] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 8 LEADER]: Raft consensus shutting down.
I20260317 16:16:33.560240 16879 raft_consensus.cc:2272] T 9527b12d0cba460ea05b4ddcf46a0568 P 7365fecf4e9e48b084df0c90eff83908 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.562856 16879 tablet_replica.cc:333] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.563619 16879 raft_consensus.cc:2243] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:16:33.564829 16879 raft_consensus.cc:2272] T aff0d35de1f84b6d9e26764809b83e10 P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.567365 16879 tablet_replica.cc:333] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.568082 16879 raft_consensus.cc:2243] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:33.568668 16879 raft_consensus.cc:2272] T 16a83877f56c4ad7be4fdfb90a68005e P 7365fecf4e9e48b084df0c90eff83908 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.571612 16879 tablet_replica.cc:333] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.572373 16879 raft_consensus.cc:2243] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:33.573020 16879 raft_consensus.cc:2272] T 995dacb7e66f4c47842e9eba5645d8c6 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.575603 16879 tablet_replica.cc:333] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.576293 16879 raft_consensus.cc:2243] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:33.576898 16879 raft_consensus.cc:2272] T 4e2510c1001e4134bf6f61fa639c3551 P 7365fecf4e9e48b084df0c90eff83908 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:33.579715 16879 tablet_replica.cc:333] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908: stopping tablet replica
I20260317 16:16:33.580407 16879 raft_consensus.cc:2243] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:33.580966 16879 raft_consensus.cc:2272] T ae772e8721354fe2bffffe3b986eaf8e P 7365fecf4e9e48b084df0c90eff83908 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:16:33.787339 17054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111)
W20260317 16:16:33.791460 17054 consensus_peers.cc:597] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 -> Peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Couldn't send request to peer 7365fecf4e9e48b084df0c90eff83908. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:16:33.883111 17054 consensus_peers.cc:597] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 -> Peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Couldn't send request to peer 7365fecf4e9e48b084df0c90eff83908. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:16:33.893020 17128 consensus_peers.cc:597] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 -> Peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Couldn't send request to peer 7365fecf4e9e48b084df0c90eff83908. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:16:33.928411 17128 consensus_peers.cc:597] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 -> Peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Couldn't send request to peer 7365fecf4e9e48b084df0c90eff83908. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:16:33.965246 17128 consensus_peers.cc:597] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 -> Peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Couldn't send request to peer 7365fecf4e9e48b084df0c90eff83908. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:16:34.011602 17054 consensus_peers.cc:597] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 -> Peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Couldn't send request to peer 7365fecf4e9e48b084df0c90eff83908. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 16:16:34.049708 16879 tablet_server.cc:195] TabletServer@127.16.123.193:0 shutdown complete.
I20260317 16:16:34.083459 16879 tablet_server.cc:178] TabletServer@127.16.123.194:0 shutting down...
I20260317 16:16:34.086648 17478 raft_consensus.cc:493] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7365fecf4e9e48b084df0c90eff83908)
I20260317 16:16:34.087218 17478 raft_consensus.cc:515] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7365fecf4e9e48b084df0c90eff83908" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 33887 } } peers { permanent_uuid: "c0bfe88236c24bc2971392e273683fd8" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 34563 } } peers { permanent_uuid: "dfb44a013db548e593d221916ed0dbd7" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 37361 } }
I20260317 16:16:34.089155 17478 leader_election.cc:290] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887), dfb44a013db548e593d221916ed0dbd7 (127.16.123.195:37361)
I20260317 16:16:34.090615 17162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d52fb95165d493db57d26439ab3b47d" candidate_uuid: "c0bfe88236c24bc2971392e273683fd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dfb44a013db548e593d221916ed0dbd7" is_pre_election: true
W20260317 16:16:34.095510 17054 leader_election.cc:336] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7365fecf4e9e48b084df0c90eff83908 (127.16.123.193:33887): Network error: Client connection negotiation failed: client connection to 127.16.123.193:33887: connect: Connection refused (error 111)
I20260317 16:16:34.095956 17054 leader_election.cc:304] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0bfe88236c24bc2971392e273683fd8; no voters: 7365fecf4e9e48b084df0c90eff83908, dfb44a013db548e593d221916ed0dbd7
I20260317 16:16:34.097638 17478 raft_consensus.cc:2749] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:16:34.116051 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:16:34.116730 16879 tablet_replica.cc:333] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.117337 16879 raft_consensus.cc:2243] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.117871 16879 raft_consensus.cc:2272] T cde56c3fe80f4f94a1981c90ca3eb089 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.119792 16879 tablet_replica.cc:333] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.120298 16879 raft_consensus.cc:2243] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:16:34.121042 16879 raft_consensus.cc:2272] T 4e2510c1001e4134bf6f61fa639c3551 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.122627 16879 tablet_replica.cc:333] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.123214 16879 raft_consensus.cc:2243] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.123625 16879 raft_consensus.cc:2272] T 16a83877f56c4ad7be4fdfb90a68005e P c0bfe88236c24bc2971392e273683fd8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.125756 16879 tablet_replica.cc:333] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.126329 16879 raft_consensus.cc:2243] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.126741 16879 raft_consensus.cc:2272] T 995dacb7e66f4c47842e9eba5645d8c6 P c0bfe88236c24bc2971392e273683fd8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.128614 16879 tablet_replica.cc:333] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.129194 16879 raft_consensus.cc:2243] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.129645 16879 raft_consensus.cc:2272] T 9527b12d0cba460ea05b4ddcf46a0568 P c0bfe88236c24bc2971392e273683fd8 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.131929 16879 tablet_replica.cc:333] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.132617 16879 raft_consensus.cc:2243] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.133412 16879 raft_consensus.cc:2272] T 7d52fb95165d493db57d26439ab3b47d P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260317 16:16:34.134303 17129 consensus_peers.cc:597] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 -> Peer c0bfe88236c24bc2971392e273683fd8 (127.16.123.194:34563): Couldn't send request to peer c0bfe88236c24bc2971392e273683fd8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:16:34.136044 16879 tablet_replica.cc:333] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.136529 16879 raft_consensus.cc:2243] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:16:34.137243 16879 raft_consensus.cc:2272] T ae772e8721354fe2bffffe3b986eaf8e P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.139138 16879 tablet_replica.cc:333] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.139685 16879 raft_consensus.cc:2243] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 5 LEADER]: Raft consensus shutting down.
I20260317 16:16:34.140537 16879 raft_consensus.cc:2272] T 94613bfab95744bca18c8c52f109387c P c0bfe88236c24bc2971392e273683fd8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.142745 16879 tablet_replica.cc:333] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8: stopping tablet replica
I20260317 16:16:34.143343 16879 raft_consensus.cc:2243] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.143990 16879 raft_consensus.cc:2272] T aff0d35de1f84b6d9e26764809b83e10 P c0bfe88236c24bc2971392e273683fd8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.173735 16879 tablet_server.cc:195] TabletServer@127.16.123.194:0 shutdown complete.
I20260317 16:16:34.193168 16879 tablet_server.cc:178] TabletServer@127.16.123.195:0 shutting down...
I20260317 16:16:34.225931 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:16:34.226583 16879 tablet_replica.cc:333] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.227355 16879 raft_consensus.cc:2243] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:16:34.228250 16879 raft_consensus.cc:2272] T cde56c3fe80f4f94a1981c90ca3eb089 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.230023 16879 tablet_replica.cc:333] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.230680 16879 raft_consensus.cc:2243] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.231474 16879 raft_consensus.cc:2272] T aff0d35de1f84b6d9e26764809b83e10 P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.233175 16879 tablet_replica.cc:333] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.233687 16879 raft_consensus.cc:2243] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.234139 16879 raft_consensus.cc:2272] T ae772e8721354fe2bffffe3b986eaf8e P dfb44a013db548e593d221916ed0dbd7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.236068 16879 tablet_replica.cc:333] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.236594 16879 raft_consensus.cc:2243] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.237190 16879 raft_consensus.cc:2272] T 4e2510c1001e4134bf6f61fa639c3551 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.238818 16879 tablet_replica.cc:333] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.239288 16879 raft_consensus.cc:2243] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:16:34.240054 16879 raft_consensus.cc:2272] T 16a83877f56c4ad7be4fdfb90a68005e P dfb44a013db548e593d221916ed0dbd7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.241659 16879 tablet_replica.cc:333] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.242115 16879 raft_consensus.cc:2243] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:16:34.243011 16879 raft_consensus.cc:2272] T 995dacb7e66f4c47842e9eba5645d8c6 P dfb44a013db548e593d221916ed0dbd7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.244652 16879 tablet_replica.cc:333] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.245137 16879 raft_consensus.cc:2243] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.245545 16879 raft_consensus.cc:2272] T 94613bfab95744bca18c8c52f109387c P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.247222 16879 tablet_replica.cc:333] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.247649 16879 raft_consensus.cc:2243] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.248080 16879 raft_consensus.cc:2272] T 7d52fb95165d493db57d26439ab3b47d P dfb44a013db548e593d221916ed0dbd7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:34.249521 16879 tablet_replica.cc:333] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7: stopping tablet replica
I20260317 16:16:34.249936 16879 raft_consensus.cc:2243] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 8 FOLLOWER]: Raft consensus shutting down.
I20260317 16:16:34.250352 16879 raft_consensus.cc:2272] T 9527b12d0cba460ea05b4ddcf46a0568 P dfb44a013db548e593d221916ed0dbd7 [term 8 FOLLOWER]: Raft consensus is shut down!
W20260317 16:16:34.301522 17184 debug-util.cc:398] Leaking SignalData structure 0x7b080016e060 after lost signal to thread 16882
W20260317 16:16:34.302294 17184 debug-util.cc:398] Leaking SignalData structure 0x7b080016e740 after lost signal to thread 16945
W20260317 16:16:34.303174 17184 debug-util.cc:398] Leaking SignalData structure 0x7b08001ebe60 after lost signal to thread 17038
W20260317 16:16:34.303902 17184 debug-util.cc:398] Leaking SignalData structure 0x7b08001d1a00 after lost signal to thread 17113
W20260317 16:16:34.304702 17184 debug-util.cc:398] Leaking SignalData structure 0x7b0800092380 after lost signal to thread 17187
W20260317 16:16:35.278694 16879 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 17184)
W20260317 16:16:36.279290 16879 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 17184)
I20260317 16:16:36.920200 16879 tablet_server.cc:195] TabletServer@127.16.123.195:0 shutdown complete.
I20260317 16:16:36.940419 16879 master.cc:561] Master@127.16.123.254:44199 shutting down...
I20260317 16:16:37.134842 16879 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:16:37.135605 16879 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:16:37.136010 16879 tablet_replica.cc:333] T 00000000000000000000000000000000 P 695516ce5dd34707b37fb75a0b69e90d: stopping tablet replica
I20260317 16:16:37.157444 16879 master.cc:583] Master@127.16.123.254:44199 shutdown complete.
7365fecf4e9e48b084df0c90eff83908 4
c0bfe88236c24bc2971392e273683fd8 4
dfb44a013db548e593d221916ed0dbd7 1
7365fecf4e9e48b084df0c90eff83908 3
c0bfe88236c24bc2971392e273683fd8 3
dfb44a013db548e593d221916ed0dbd7 3
7365fecf4e9e48b084df0c90eff83908 0
c0bfe88236c24bc2971392e273683fd8 8
dfb44a013db548e593d221916ed0dbd7 1
7365fecf4e9e48b084df0c90eff83908 3
c0bfe88236c24bc2971392e273683fd8 3
dfb44a013db548e593d221916ed0dbd7 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (36282 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260317 16:16:37.199430 16879 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.123.254:40161
I20260317 16:16:37.200582 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:37.206665 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:37.207019 17492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:37.207805 17494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:37.209019 16879 server_base.cc:1060] running on GCE node
I20260317 16:16:37.210000 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:37.210260 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:37.210418 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764197210403 us; error 0 us; skew 500 ppm
I20260317 16:16:37.211027 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:37.213578 16879 webserver.cc:492] Webserver started at http://127.16.123.254:39091/ using document root <none> and password file <none>
I20260317 16:16:37.214331 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:37.214602 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:37.215240 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:37.216420 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/master-0-root/instance:
uuid: "dc2066b9378844879b54182e6a8f8614"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.220925 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260317 16:16:37.224213 17499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.225116 16879 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260317 16:16:37.225351 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/master-0-root
uuid: "dc2066b9378844879b54182e6a8f8614"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.225589 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:37.251122 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:37.252507 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:37.294030 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.254:40161
I20260317 16:16:37.294142 17550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.254:40161 every 8 connection(s)
I20260317 16:16:37.298309 17551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.307832 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614: Bootstrap starting.
I20260317 16:16:37.312562 17551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:37.316911 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614: No bootstrap required, opened a new log
I20260317 16:16:37.319046 17551 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2066b9378844879b54182e6a8f8614" member_type: VOTER }
I20260317 16:16:37.319442 17551 raft_consensus.cc:385] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:37.319684 17551 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2066b9378844879b54182e6a8f8614, State: Initialized, Role: FOLLOWER
I20260317 16:16:37.320325 17551 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2066b9378844879b54182e6a8f8614" member_type: VOTER }
I20260317 16:16:37.320780 17551 raft_consensus.cc:399] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 16:16:37.321009 17551 raft_consensus.cc:493] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 16:16:37.321317 17551 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:37.325441 17551 raft_consensus.cc:515] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2066b9378844879b54182e6a8f8614" member_type: VOTER }
I20260317 16:16:37.326009 17551 leader_election.cc:304] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2066b9378844879b54182e6a8f8614; no voters:
I20260317 16:16:37.327339 17551 leader_election.cc:290] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 16:16:37.327761 17554 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:37.329422 17554 raft_consensus.cc:697] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 1 LEADER]: Becoming Leader. State: Replica: dc2066b9378844879b54182e6a8f8614, State: Running, Role: LEADER
I20260317 16:16:37.330134 17554 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2066b9378844879b54182e6a8f8614" member_type: VOTER }
I20260317 16:16:37.330423 17551 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 16:16:37.335031 17556 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2066b9378844879b54182e6a8f8614. Latest consensus state: current_term: 1 leader_uuid: "dc2066b9378844879b54182e6a8f8614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2066b9378844879b54182e6a8f8614" member_type: VOTER } }
I20260317 16:16:37.335660 17556 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [sys.catalog]: This master's current role is: LEADER
I20260317 16:16:37.336807 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2066b9378844879b54182e6a8f8614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2066b9378844879b54182e6a8f8614" member_type: VOTER } }
I20260317 16:16:37.337522 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [sys.catalog]: This master's current role is: LEADER
I20260317 16:16:37.339156 17561 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 16:16:37.347004 17561 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 16:16:37.348134 16879 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 16:16:37.356885 17561 catalog_manager.cc:1357] Generated new cluster ID: 9dcb3771e71941f98aeeaad0d9eaa7a0
I20260317 16:16:37.357293 17561 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 16:16:37.372011 17561 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 16:16:37.373404 17561 catalog_manager.cc:1514] Loading token signing keys...
I20260317 16:16:37.388677 17561 catalog_manager.cc:6044] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614: Generated new TSK 0
I20260317 16:16:37.389303 17561 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 16:16:37.415215 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:37.422588 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:37.423789 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:37.426162 16879 server_base.cc:1060] running on GCE node
W20260317 16:16:37.427352 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:37.428531 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:37.428745 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:37.429023 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764197429006 us; error 0 us; skew 500 ppm
I20260317 16:16:37.429692 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:37.433439 16879 webserver.cc:492] Webserver started at http://127.16.123.193:44747/ using document root <none> and password file <none>
I20260317 16:16:37.434000 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:37.434214 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:37.434448 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:37.436054 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-0-root/instance:
uuid: "9cb1383296ba48d6a985080124b4ad1e"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.442412 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260317 16:16:37.447590 17580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.448969 16879 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260317 16:16:37.449316 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-0-root
uuid: "9cb1383296ba48d6a985080124b4ad1e"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.449596 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:37.483758 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:37.485225 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:37.487087 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:16:37.489723 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:16:37.489990 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.490290 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:16:37.490653 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.542687 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.193:35607
I20260317 16:16:37.542927 17642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.193:35607 every 8 connection(s)
I20260317 16:16:37.549252 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:37.557274 17647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:37.558478 17648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:37.562708 17650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:37.564124 17643 heartbeater.cc:344] Connected to a master server at 127.16.123.254:40161
I20260317 16:16:37.564195 16879 server_base.cc:1060] running on GCE node
I20260317 16:16:37.564671 17643 heartbeater.cc:461] Registering TS with master...
I20260317 16:16:37.565439 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260317 16:16:37.565431 17643 heartbeater.cc:507] Master 127.16.123.254:40161 requested a full tablet report, sending...
W20260317 16:16:37.565819 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:37.566109 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764197566086 us; error 0 us; skew 500 ppm
I20260317 16:16:37.566952 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:37.568609 17516 ts_manager.cc:194] Registered new tserver with Master: 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607)
I20260317 16:16:37.570000 16879 webserver.cc:492] Webserver started at http://127.16.123.194:43095/ using document root <none> and password file <none>
I20260317 16:16:37.570699 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:37.570979 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:37.571308 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:37.571300 17516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59472
I20260317 16:16:37.573112 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-1-root/instance:
uuid: "06e13d9b08e141a58988aa72744d54be"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.577817 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260317 16:16:37.581522 17655 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.582567 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260317 16:16:37.583035 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-1-root
uuid: "06e13d9b08e141a58988aa72744d54be"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.583323 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:37.596977 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:37.598246 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:37.599972 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:16:37.602320 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:16:37.602507 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.602820 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:16:37.603178 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260317 16:16:37.670382 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.194:32955
I20260317 16:16:37.670755 17717 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.194:32955 every 8 connection(s)
I20260317 16:16:37.676084 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:16:37.685316 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:16:37.688088 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:37.692396 17718 heartbeater.cc:344] Connected to a master server at 127.16.123.254:40161
I20260317 16:16:37.693073 17718 heartbeater.cc:461] Registering TS with master...
W20260317 16:16:37.693971 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:16:37.694192 17718 heartbeater.cc:507] Master 127.16.123.254:40161 requested a full tablet report, sending...
I20260317 16:16:37.695072 16879 server_base.cc:1060] running on GCE node
I20260317 16:16:37.696870 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:16:37.697160 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:16:37.697365 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764197697337 us; error 0 us; skew 500 ppm
I20260317 16:16:37.697501 17516 ts_manager.cc:194] Registered new tserver with Master: 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:37.698078 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:16:37.699769 17516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59486
I20260317 16:16:37.701350 16879 webserver.cc:492] Webserver started at http://127.16.123.195:42799/ using document root <none> and password file <none>
I20260317 16:16:37.701889 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:16:37.702105 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:16:37.702391 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:16:37.703625 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-2-root/instance:
uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.709740 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260317 16:16:37.714193 17729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.715446 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260317 16:16:37.716192 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-2-root
uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
format_stamp: "Formatted at 2026-03-17 16:16:37 on dist-test-slave-kqc6"
I20260317 16:16:37.716501 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1773764160731778-16879-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:16:37.759578 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:16:37.760773 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:16:37.762177 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:16:37.764431 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:16:37.764658 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.764919 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:16:37.765087 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:16:37.816805 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.195:33255
I20260317 16:16:37.817131 17791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.195:33255 every 8 connection(s)
I20260317 16:16:37.832072 17792 heartbeater.cc:344] Connected to a master server at 127.16.123.254:40161
I20260317 16:16:37.832954 17792 heartbeater.cc:461] Registering TS with master...
I20260317 16:16:37.833806 17792 heartbeater.cc:507] Master 127.16.123.254:40161 requested a full tablet report, sending...
I20260317 16:16:37.837256 17516 ts_manager.cc:194] Registered new tserver with Master: c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:37.838406 16879 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018431911s
I20260317 16:16:37.838721 17516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59488
I20260317 16:16:37.871644 17516 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59492:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 16:16:37.874686 17516 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 16:16:37.952425 17683 tablet_service.cc:1511] Processing CreateTablet for tablet cbad27cc136042689fa5425accbb4419 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260317 16:16:37.953045 17681 tablet_service.cc:1511] Processing CreateTablet for tablet 24d1d1ab50904f4a936aaefffcdfd42d (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260317 16:16:37.954448 17683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbad27cc136042689fa5425accbb4419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.954216 17680 tablet_service.cc:1511] Processing CreateTablet for tablet 8f286f6866fc4d37a61f21afe7b892fe (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260317 16:16:37.955631 17682 tablet_service.cc:1511] Processing CreateTablet for tablet abad741d31214ee4b0e8b8a3bbb2ead7 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260317 16:16:37.956867 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 5ecf40a1e6da4628b84dbfb94499ff15 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260317 16:16:37.958395 17680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f286f6866fc4d37a61f21afe7b892fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.960238 17757 tablet_service.cc:1511] Processing CreateTablet for tablet cbad27cc136042689fa5425accbb4419 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260317 16:16:37.961541 17757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbad27cc136042689fa5425accbb4419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.963567 17682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abad741d31214ee4b0e8b8a3bbb2ead7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.962623 17756 tablet_service.cc:1511] Processing CreateTablet for tablet abad741d31214ee4b0e8b8a3bbb2ead7 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260317 16:16:37.964823 17756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abad741d31214ee4b0e8b8a3bbb2ead7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.969026 17678 tablet_service.cc:1511] Processing CreateTablet for tablet d4c2c3cb2259449ab45cc2db20e46ef6 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260317 16:16:37.969451 17677 tablet_service.cc:1511] Processing CreateTablet for tablet 8b1a38ac52d14630bd9aec18edd53793 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260317 16:16:37.970433 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c2c3cb2259449ab45cc2db20e46ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.971112 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ecf40a1e6da4628b84dbfb94499ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.974624 17676 tablet_service.cc:1511] Processing CreateTablet for tablet 07a4694aee514c9e8f65ebab0a511dc3 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260317 16:16:37.976332 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a4694aee514c9e8f65ebab0a511dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.980453 17675 tablet_service.cc:1511] Processing CreateTablet for tablet 38527a3689314a1aa8e83ff19e123d60 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260317 16:16:37.981899 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38527a3689314a1aa8e83ff19e123d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.983744 17755 tablet_service.cc:1511] Processing CreateTablet for tablet 24d1d1ab50904f4a936aaefffcdfd42d (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260317 16:16:37.984102 17608 tablet_service.cc:1511] Processing CreateTablet for tablet cbad27cc136042689fa5425accbb4419 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260317 16:16:37.985306 17755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d1d1ab50904f4a936aaefffcdfd42d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.985795 17608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbad27cc136042689fa5425accbb4419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.988603 17681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d1d1ab50904f4a936aaefffcdfd42d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.988804 17754 tablet_service.cc:1511] Processing CreateTablet for tablet 8f286f6866fc4d37a61f21afe7b892fe (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260317 16:16:37.990231 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f286f6866fc4d37a61f21afe7b892fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.991192 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1a38ac52d14630bd9aec18edd53793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.992074 17601 tablet_service.cc:1511] Processing CreateTablet for tablet 07a4694aee514c9e8f65ebab0a511dc3 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260317 16:16:37.992494 17607 tablet_service.cc:1511] Processing CreateTablet for tablet abad741d31214ee4b0e8b8a3bbb2ead7 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260317 16:16:37.993605 17752 tablet_service.cc:1511] Processing CreateTablet for tablet d4c2c3cb2259449ab45cc2db20e46ef6 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260317 16:16:37.994797 17753 tablet_service.cc:1511] Processing CreateTablet for tablet 5ecf40a1e6da4628b84dbfb94499ff15 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260317 16:16:37.995800 17606 tablet_service.cc:1511] Processing CreateTablet for tablet 24d1d1ab50904f4a936aaefffcdfd42d (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260317 16:16:37.996728 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c2c3cb2259449ab45cc2db20e46ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.997112 17605 tablet_service.cc:1511] Processing CreateTablet for tablet 8f286f6866fc4d37a61f21afe7b892fe (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260317 16:16:37.999972 17604 tablet_service.cc:1511] Processing CreateTablet for tablet 5ecf40a1e6da4628b84dbfb94499ff15 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260317 16:16:38.001209 17751 tablet_service.cc:1511] Processing CreateTablet for tablet 8b1a38ac52d14630bd9aec18edd53793 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260317 16:16:38.002576 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1a38ac52d14630bd9aec18edd53793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.002494 17603 tablet_service.cc:1511] Processing CreateTablet for tablet d4c2c3cb2259449ab45cc2db20e46ef6 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260317 16:16:38.005064 17602 tablet_service.cc:1511] Processing CreateTablet for tablet 8b1a38ac52d14630bd9aec18edd53793 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260317 16:16:38.007982 17600 tablet_service.cc:1511] Processing CreateTablet for tablet 38527a3689314a1aa8e83ff19e123d60 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260317 16:16:38.015439 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ecf40a1e6da4628b84dbfb94499ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:37.993626 17601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a4694aee514c9e8f65ebab0a511dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.019135 17603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c2c3cb2259449ab45cc2db20e46ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.023224 17605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f286f6866fc4d37a61f21afe7b892fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.027280 17599 tablet_service.cc:1511] Processing CreateTablet for tablet e38f8a85a03e423a8924458110a6cbf0 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260317 16:16:38.028766 17599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38f8a85a03e423a8924458110a6cbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.031384 17602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b1a38ac52d14630bd9aec18edd53793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.032845 17604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ecf40a1e6da4628b84dbfb94499ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.034332 17607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abad741d31214ee4b0e8b8a3bbb2ead7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.037433 17600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38527a3689314a1aa8e83ff19e123d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.044440 17750 tablet_service.cc:1511] Processing CreateTablet for tablet 07a4694aee514c9e8f65ebab0a511dc3 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260317 16:16:38.045964 17750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a4694aee514c9e8f65ebab0a511dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.051472 17606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d1d1ab50904f4a936aaefffcdfd42d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.071417 17678 tablet_service.cc:1511] Processing CreateTablet for tablet e38f8a85a03e423a8924458110a6cbf0 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260317 16:16:38.072723 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38f8a85a03e423a8924458110a6cbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.101750 17757 tablet_service.cc:1511] Processing CreateTablet for tablet 38527a3689314a1aa8e83ff19e123d60 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260317 16:16:38.111886 17808 tablet_bootstrap.cc:492] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.119140 17757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38527a3689314a1aa8e83ff19e123d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.121040 17749 tablet_service.cc:1511] Processing CreateTablet for tablet e38f8a85a03e423a8924458110a6cbf0 (DEFAULT_TABLE table=test-workload [id=09bb65e8489f4e9b9eef5921b5488d47]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260317 16:16:38.122699 17749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38f8a85a03e423a8924458110a6cbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:16:38.139814 17807 tablet_bootstrap.cc:492] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.173360 17808 tablet_bootstrap.cc:654] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.199096 17808 tablet_bootstrap.cc:492] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.199584 17808 ts_tablet_manager.cc:1403] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.009s
I20260317 16:16:38.203173 17807 tablet_bootstrap.cc:654] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.203372 17808 raft_consensus.cc:359] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.204447 17808 raft_consensus.cc:385] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.204764 17808 raft_consensus.cc:740] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.205631 17808 consensus_queue.cc:260] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.207017 17809 tablet_bootstrap.cc:492] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.215478 17807 tablet_bootstrap.cc:492] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.216313 17807 ts_tablet_manager.cc:1403] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.004s
I20260317 16:16:38.219506 17807 raft_consensus.cc:359] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.220326 17807 raft_consensus.cc:385] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.220338 17809 tablet_bootstrap.cc:654] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.220645 17807 raft_consensus.cc:740] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.221603 17807 consensus_queue.cc:260] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.223874 17792 heartbeater.cc:499] Master 127.16.123.254:40161 was elected leader, sending a full tablet report...
I20260317 16:16:38.225750 17807 ts_tablet_manager.cc:1434] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260317 16:16:38.228622 17807 tablet_bootstrap.cc:492] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.228579 17809 tablet_bootstrap.cc:492] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.229990 17812 raft_consensus.cc:493] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.230556 17809 ts_tablet_manager.cc:1403] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.007s
I20260317 16:16:38.230597 17812 raft_consensus.cc:515] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.236806 17812 leader_election.cc:290] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607)
I20260317 16:16:38.238420 17808 ts_tablet_manager.cc:1434] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.039s user 0.008s sys 0.002s
I20260317 16:16:38.239573 17718 heartbeater.cc:499] Master 127.16.123.254:40161 was elected leader, sending a full tablet report...
I20260317 16:16:38.240365 17808 tablet_bootstrap.cc:492] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.233727 17809 raft_consensus.cc:359] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.241158 17809 raft_consensus.cc:385] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.241623 17809 raft_consensus.cc:740] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.244158 17809 consensus_queue.cc:260] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.249962 17808 tablet_bootstrap.cc:654] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.254805 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
W20260317 16:16:38.257440 17733 leader_election.cc:343] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.260780 17807 tablet_bootstrap.cc:654] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.276685 17643 heartbeater.cc:499] Master 127.16.123.254:40161 was elected leader, sending a full tablet report...
I20260317 16:16:38.277544 17809 ts_tablet_manager.cc:1434] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.047s user 0.015s sys 0.005s
I20260317 16:16:38.278530 17809 tablet_bootstrap.cc:492] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.276697 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
W20260317 16:16:38.282600 17732 leader_election.cc:343] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.283155 17732 leader_election.cc:304] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:38.284026 17812 raft_consensus.cc:2749] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:38.285305 17809 tablet_bootstrap.cc:654] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.286963 17807 tablet_bootstrap.cc:492] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.287468 17807 ts_tablet_manager.cc:1403] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.002s
I20260317 16:16:38.291677 17807 raft_consensus.cc:359] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.292665 17807 raft_consensus.cc:385] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.293067 17807 raft_consensus.cc:740] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.294219 17807 consensus_queue.cc:260] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.296859 17807 ts_tablet_manager.cc:1434] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260317 16:16:38.297588 17809 tablet_bootstrap.cc:492] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.297948 17807 tablet_bootstrap.cc:492] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.298122 17809 ts_tablet_manager.cc:1403] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
W20260317 16:16:38.298812 17644 tablet.cc:2379] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 16:16:38.301491 17809 raft_consensus.cc:359] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.302486 17809 raft_consensus.cc:385] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.302888 17809 raft_consensus.cc:740] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.303823 17809 consensus_queue.cc:260] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.307022 17809 ts_tablet_manager.cc:1434] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260317 16:16:38.307478 17807 tablet_bootstrap.cc:654] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.308066 17809 tablet_bootstrap.cc:492] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.312726 17808 tablet_bootstrap.cc:492] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.313416 17808 ts_tablet_manager.cc:1403] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.049s
I20260317 16:16:38.315788 17809 tablet_bootstrap.cc:654] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.317679 17807 tablet_bootstrap.cc:492] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.317427 17808 raft_consensus.cc:359] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.318315 17807 ts_tablet_manager.cc:1403] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20260317 16:16:38.318300 17808 raft_consensus.cc:385] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.319036 17808 raft_consensus.cc:740] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.319955 17808 consensus_queue.cc:260] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.324178 17808 ts_tablet_manager.cc:1434] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260317 16:16:38.325217 17808 tablet_bootstrap.cc:492] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.327934 17809 tablet_bootstrap.cc:492] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.328526 17809 ts_tablet_manager.cc:1403] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20260317 16:16:38.331835 17808 tablet_bootstrap.cc:654] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.331452 17809 raft_consensus.cc:359] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.332463 17809 raft_consensus.cc:385] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.332814 17809 raft_consensus.cc:740] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.332947 17807 raft_consensus.cc:359] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.333673 17807 raft_consensus.cc:385] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.334048 17807 raft_consensus.cc:740] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.333765 17809 consensus_queue.cc:260] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.334784 17807 consensus_queue.cc:260] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.336529 17809 ts_tablet_manager.cc:1434] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260317 16:16:38.337647 17809 tablet_bootstrap.cc:492] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.339242 17807 ts_tablet_manager.cc:1434] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260317 16:16:38.340713 17807 tablet_bootstrap.cc:492] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
W20260317 16:16:38.341575 17564 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:16:38.343837 17808 tablet_bootstrap.cc:492] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.344393 17808 ts_tablet_manager.cc:1403] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20260317 16:16:38.346786 17809 tablet_bootstrap.cc:654] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.347426 17808 raft_consensus.cc:359] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.348356 17808 raft_consensus.cc:385] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.348708 17808 raft_consensus.cc:740] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.349581 17808 consensus_queue.cc:260] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.351780 17807 tablet_bootstrap.cc:654] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.352522 17809 tablet_bootstrap.cc:492] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.353049 17809 ts_tablet_manager.cc:1403] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260317 16:16:38.355033 17808 ts_tablet_manager.cc:1434] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260317 16:16:38.356137 17808 tablet_bootstrap.cc:492] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.355958 17809 raft_consensus.cc:359] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.356688 17809 raft_consensus.cc:385] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.357071 17809 raft_consensus.cc:740] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.357988 17809 consensus_queue.cc:260] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.360754 17809 ts_tablet_manager.cc:1434] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260317 16:16:38.361841 17809 tablet_bootstrap.cc:492] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.363266 17808 tablet_bootstrap.cc:654] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.370625 17807 tablet_bootstrap.cc:492] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.371064 17809 tablet_bootstrap.cc:654] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.371316 17807 ts_tablet_manager.cc:1403] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.004s
I20260317 16:16:38.374675 17807 raft_consensus.cc:359] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.376600 17807 raft_consensus.cc:385] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.376881 17807 raft_consensus.cc:740] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.379886 17807 consensus_queue.cc:260] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.381099 17809 tablet_bootstrap.cc:492] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.381599 17809 ts_tablet_manager.cc:1403] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260317 16:16:38.381764 17808 tablet_bootstrap.cc:492] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.382242 17808 ts_tablet_manager.cc:1403] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.005s
I20260317 16:16:38.382673 17807 ts_tablet_manager.cc:1434] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260317 16:16:38.384325 17807 tablet_bootstrap.cc:492] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.385167 17809 raft_consensus.cc:359] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.385942 17809 raft_consensus.cc:385] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.385608 17808 raft_consensus.cc:359] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.386266 17809 raft_consensus.cc:740] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.386674 17808 raft_consensus.cc:385] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.387164 17812 raft_consensus.cc:493] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.387147 17809 consensus_queue.cc:260] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.387962 17816 raft_consensus.cc:493] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.388466 17816 raft_consensus.cc:515] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.387743 17812 raft_consensus.cc:515] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.393657 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:38.394686 17807 tablet_bootstrap.cc:654] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
W20260317 16:16:38.399012 17733 leader_election.cc:343] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.399775 17816 leader_election.cc:290] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.402530 17809 ts_tablet_manager.cc:1434] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20260317 16:16:38.403939 17809 tablet_bootstrap.cc:492] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.387135 17808 raft_consensus.cc:740] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.410980 17807 tablet_bootstrap.cc:492] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.411608 17809 tablet_bootstrap.cc:654] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.411578 17807 ts_tablet_manager.cc:1403] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.016s
I20260317 16:16:38.412043 17808 consensus_queue.cc:260] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.413305 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.413961 17693 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 0.
I20260317 16:16:38.414674 17808 ts_tablet_manager.cc:1434] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260317 16:16:38.415426 17732 leader_election.cc:304] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:38.415814 17808 tablet_bootstrap.cc:492] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.416579 17812 leader_election.cc:290] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.417143 17812 raft_consensus.cc:2804] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.417462 17812 raft_consensus.cc:493] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.417805 17812 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.480118 17827 raft_consensus.cc:493] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.480841 17827 raft_consensus.cc:515] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.484711 17812 raft_consensus.cc:515] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.502060 17816 raft_consensus.cc:493] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.502779 17816 raft_consensus.cc:515] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.510277 17814 raft_consensus.cc:493] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.511044 17814 raft_consensus.cc:515] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.517441 17832 raft_consensus.cc:493] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.518028 17832 raft_consensus.cc:515] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.517344 17812 leader_election.cc:290] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.520334 17832 leader_election.cc:290] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.515038 17814 leader_election.cc:290] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.524000 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:38.524407 17807 raft_consensus.cc:359] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.525097 17807 raft_consensus.cc:385] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.525379 17807 raft_consensus.cc:740] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.522746 17827 leader_election.cc:290] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.526047 17827 raft_consensus.cc:493] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.526206 17807 consensus_queue.cc:260] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.526471 17827 raft_consensus.cc:515] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.528834 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.529433 17693 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.530829 17809 tablet_bootstrap.cc:492] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.531315 17809 ts_tablet_manager.cc:1403] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.025s
I20260317 16:16:38.534027 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:38.534188 17809 raft_consensus.cc:359] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.534683 17618 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 0.
I20260317 16:16:38.534940 17809 raft_consensus.cc:385] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.535275 17809 raft_consensus.cc:740] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.535486 17816 leader_election.cc:290] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
W20260317 16:16:38.532624 17733 leader_election.cc:343] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.536058 17809 consensus_queue.cc:260] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.536593 17733 leader_election.cc:304] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:38.537611 17832 raft_consensus.cc:2804] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.538021 17832 raft_consensus.cc:493] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.538127 17693 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:38.538470 17832 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.538679 17809 ts_tablet_manager.cc:1434] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 16:16:38.540292 17827 leader_election.cc:290] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.540833 17809 tablet_bootstrap.cc:492] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.549479 17807 ts_tablet_manager.cc:1434] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.137s user 0.010s sys 0.017s
I20260317 16:16:38.553557 17809 tablet_bootstrap.cc:654] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.557523 17807 tablet_bootstrap.cc:492] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.565161 17808 tablet_bootstrap.cc:654] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.572710 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.573683 17832 raft_consensus.cc:515] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.578042 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:38.578253 17832 leader_election.cc:290] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.578653 17618 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.581792 17732 leader_election.cc:304] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 9cb1383296ba48d6a985080124b4ad1e
W20260317 16:16:38.583110 17732 leader_election.cc:343] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.584602 17807 tablet_bootstrap.cc:654] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.584915 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.588033 17618 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:38.590444 17692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.591449 17733 leader_election.cc:304] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:38.590848 17690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38527a3689314a1aa8e83ff19e123d60" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.586236 17832 raft_consensus.cc:2804] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:38.593582 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.593972 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.594542 17812 raft_consensus.cc:2804] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election won for term 1
W20260317 16:16:38.595072 17583 leader_election.cc:343] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.596804 17691 raft_consensus.cc:2468] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 0.
I20260317 16:16:38.597370 17766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:38.602268 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38527a3689314a1aa8e83ff19e123d60" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:38.604620 17765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:38.605518 17765 raft_consensus.cc:2468] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 0.
I20260317 16:16:38.599133 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
W20260317 16:16:38.601426 17583 leader_election.cc:343] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.608422 17583 leader_election.cc:304] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:38.596079 17693 raft_consensus.cc:2468] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 0.
I20260317 16:16:38.612955 17835 raft_consensus.cc:2804] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.614224 17835 raft_consensus.cc:493] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.614794 17835 raft_consensus.cc:3060] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.617944 17583 leader_election.cc:304] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:38.619362 17827 raft_consensus.cc:2804] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.620182 17827 raft_consensus.cc:493] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.620640 17827 raft_consensus.cc:3060] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:16:38.621929 17582 leader_election.cc:343] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.624291 17582 leader_election.cc:304] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e; no voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8
W20260317 16:16:38.623023 17732 leader_election.cc:343] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.649904 17816 raft_consensus.cc:2749] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:38.651547 17827 raft_consensus.cc:515] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
W20260317 16:16:38.652061 17582 leader_election.cc:343] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.653051 17582 leader_election.cc:304] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e; no voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:38.654176 17816 raft_consensus.cc:2749] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:38.655164 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.655918 17693 raft_consensus.cc:3060] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:16:38.660714 17582 leader_election.cc:343] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.662552 17841 raft_consensus.cc:493] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.661391 17807 tablet_bootstrap.cc:492] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.663174 17841 raft_consensus.cc:515] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.663833 17807 ts_tablet_manager.cc:1403] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.113s user 0.019s sys 0.025s
I20260317 16:16:38.665426 17841 leader_election.cc:290] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.666548 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.666917 17832 raft_consensus.cc:697] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:38.667253 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:38.668057 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
W20260317 16:16:38.668738 17733 leader_election.cc:343] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.668111 17832 consensus_queue.cc:237] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.667248 17691 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 0.
I20260317 16:16:38.668908 17618 raft_consensus.cc:2393] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06e13d9b08e141a58988aa72744d54be in current term 1: Already voted for candidate 9cb1383296ba48d6a985080124b4ad1e in this term.
I20260317 16:16:38.670692 17732 leader_election.cc:304] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:38.670934 17807 raft_consensus.cc:359] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.671613 17807 raft_consensus.cc:385] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.671934 17807 raft_consensus.cc:740] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.672209 17842 raft_consensus.cc:2804] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.672581 17809 tablet_bootstrap.cc:492] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.672737 17842 raft_consensus.cc:493] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.672914 17807 consensus_queue.cc:260] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.673300 17809 ts_tablet_manager.cc:1403] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.133s user 0.014s sys 0.031s
I20260317 16:16:38.673264 17842 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.675700 17807 ts_tablet_manager.cc:1434] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20260317 16:16:38.676234 17809 raft_consensus.cc:359] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.676687 17807 tablet_bootstrap.cc:492] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.676724 17693 raft_consensus.cc:2468] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 1.
I20260317 16:16:38.676970 17809 raft_consensus.cc:385] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.677382 17809 raft_consensus.cc:740] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.678011 17583 leader_election.cc:304] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:38.678228 17809 consensus_queue.cc:260] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.678915 17816 raft_consensus.cc:2804] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:38.679438 17812 raft_consensus.cc:697] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:38.684759 17835 raft_consensus.cc:515] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.699259 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
W20260317 16:16:38.700697 17582 leader_election.cc:343] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.683123 17812 consensus_queue.cc:237] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.701449 17816 raft_consensus.cc:697] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:38.701593 17827 leader_election.cc:290] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.705979 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.707924 17693 raft_consensus.cc:3060] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.708683 17808 tablet_bootstrap.cc:492] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.708457 17816 consensus_queue.cc:237] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.709146 17808 ts_tablet_manager.cc:1403] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.294s user 0.010s sys 0.012s
I20260317 16:16:38.711637 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:38.712438 17767 raft_consensus.cc:3060] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.706111 17835 leader_election.cc:290] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.715699 17808 raft_consensus.cc:359] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.716326 17807 tablet_bootstrap.cc:654] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.716467 17808 raft_consensus.cc:385] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.716770 17808 raft_consensus.cc:740] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.716987 17693 raft_consensus.cc:2468] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 1.
I20260317 16:16:38.712474 17844 raft_consensus.cc:493] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.718230 17583 leader_election.cc:304] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:38.719185 17827 raft_consensus.cc:2804] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:38.718142 17844 raft_consensus.cc:515] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.721369 17765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:38.722239 17809 ts_tablet_manager.cc:1434] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.048s user 0.010s sys 0.017s
W20260317 16:16:38.722863 17657 leader_election.cc:343] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.723382 17657 leader_election.cc:304] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be; no voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:38.724656 17814 raft_consensus.cc:2749] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:38.722892 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.719302 17808 consensus_queue.cc:260] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.727710 17808 ts_tablet_manager.cc:1434] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20260317 16:16:38.729736 17765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:38.730376 17765 raft_consensus.cc:2468] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 0.
W20260317 16:16:38.731179 17583 leader_election.cc:343] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.732630 17582 leader_election.cc:304] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:38.733681 17843 raft_consensus.cc:2804] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.734197 17843 raft_consensus.cc:493] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.734689 17843 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.736703 17842 raft_consensus.cc:515] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.737381 17809 tablet_bootstrap.cc:492] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.719683 17827 raft_consensus.cc:697] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:38.745311 17843 raft_consensus.cc:515] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.734738 17844 leader_election.cc:290] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.747784 17808 tablet_bootstrap.cc:492] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.747254 17827 consensus_queue.cc:237] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.749969 17809 tablet_bootstrap.cc:654] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.756534 17808 tablet_bootstrap.cc:654] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.739200 17767 raft_consensus.cc:2468] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 1.
I20260317 16:16:38.757370 17842 leader_election.cc:290] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.753801 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:38.760533 17618 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.754213 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.763118 17693 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.768224 17514 catalog_manager.cc:5671] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:38.769775 17514 catalog_manager.cc:5671] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:38.778466 17618 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:38.779327 17693 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:38.780525 17732 leader_election.cc:304] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:38.781463 17812 raft_consensus.cc:2804] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:38.782030 17812 raft_consensus.cc:697] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:38.782888 17812 consensus_queue.cc:237] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.786444 17512 catalog_manager.cc:5671] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:38.790968 17843 leader_election.cc:290] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.792052 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:38.792686 17767 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.793018 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.794384 17808 tablet_bootstrap.cc:492] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.795092 17808 ts_tablet_manager.cc:1403] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.032s
W20260317 16:16:38.795413 17583 leader_election.cc:343] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.798313 17767 raft_consensus.cc:2468] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 1.
I20260317 16:16:38.799549 17582 leader_election.cc:304] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:38.799377 17808 raft_consensus.cc:359] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.800458 17808 raft_consensus.cc:385] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.800709 17827 raft_consensus.cc:2804] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:38.790375 17807 tablet_bootstrap.cc:492] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.803846 17807 ts_tablet_manager.cc:1403] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.127s user 0.011s sys 0.009s
I20260317 16:16:38.804579 17809 tablet_bootstrap.cc:492] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.805033 17809 ts_tablet_manager.cc:1403] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.006s
I20260317 16:16:38.800845 17808 raft_consensus.cc:740] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.801414 17827 raft_consensus.cc:697] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:38.806254 17808 consensus_queue.cc:260] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.806337 17827 consensus_queue.cc:237] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.807178 17807 raft_consensus.cc:359] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.807950 17807 raft_consensus.cc:385] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.808283 17807 raft_consensus.cc:740] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.809041 17809 raft_consensus.cc:359] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.809041 17807 consensus_queue.cc:260] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.809847 17809 raft_consensus.cc:385] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.810261 17809 raft_consensus.cc:740] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.810950 17809 consensus_queue.cc:260] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.812973 17807 ts_tablet_manager.cc:1434] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
I20260317 16:16:38.813004 17809 ts_tablet_manager.cc:1434] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260317 16:16:38.814551 17807 tablet_bootstrap.cc:492] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.814807 17809 tablet_bootstrap.cc:492] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.813082 17808 ts_tablet_manager.cc:1434] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260317 16:16:38.815979 17808 tablet_bootstrap.cc:492] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.823558 17807 tablet_bootstrap.cc:654] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.824818 17809 tablet_bootstrap.cc:654] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.840871 17808 tablet_bootstrap.cc:654] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.842787 17807 tablet_bootstrap.cc:492] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.843279 17807 ts_tablet_manager.cc:1403] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20260317 16:16:38.845744 17807 raft_consensus.cc:359] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.846410 17807 raft_consensus.cc:385] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.846683 17807 raft_consensus.cc:740] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.847357 17807 consensus_queue.cc:260] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.849961 17807 ts_tablet_manager.cc:1434] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260317 16:16:38.849967 17809 tablet_bootstrap.cc:492] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.851298 17809 ts_tablet_manager.cc:1403] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.004s
I20260317 16:16:38.853786 17809 raft_consensus.cc:359] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.854472 17809 raft_consensus.cc:385] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.854790 17809 raft_consensus.cc:740] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.854722 17514 catalog_manager.cc:5671] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:38.855537 17809 consensus_queue.cc:260] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.858201 17808 tablet_bootstrap.cc:492] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.858711 17807 tablet_bootstrap.cc:492] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.858743 17808 ts_tablet_manager.cc:1403] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.014s
I20260317 16:16:38.862231 17809 ts_tablet_manager.cc:1434] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260317 16:16:38.861801 17808 raft_consensus.cc:359] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.862784 17808 raft_consensus.cc:385] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.863199 17809 tablet_bootstrap.cc:492] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e: Bootstrap starting.
I20260317 16:16:38.863170 17808 raft_consensus.cc:740] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.864297 17808 consensus_queue.cc:260] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.864916 17807 tablet_bootstrap.cc:654] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.866974 17808 ts_tablet_manager.cc:1434] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260317 16:16:38.866884 17512 catalog_manager.cc:5671] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:38.868444 17808 tablet_bootstrap.cc:492] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.868770 17512 catalog_manager.cc:5671] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:38.872824 17827 raft_consensus.cc:493] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.873366 17827 raft_consensus.cc:515] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.875391 17827 leader_election.cc:290] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.878329 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.878525 17808 tablet_bootstrap.cc:654] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.879087 17693 raft_consensus.cc:2468] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 0.
I20260317 16:16:38.880065 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:38.880693 17767 raft_consensus.cc:2468] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 0.
I20260317 16:16:38.882727 17583 leader_election.cc:304] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:38.891867 17827 raft_consensus.cc:2804] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.892381 17827 raft_consensus.cc:493] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.892737 17827 raft_consensus.cc:3060] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.901319 17809 tablet_bootstrap.cc:654] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.907420 17827 raft_consensus.cc:515] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.910197 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:38.910925 17693 raft_consensus.cc:3060] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.912242 17807 tablet_bootstrap.cc:492] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.913010 17807 ts_tablet_manager.cc:1403] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.062s user 0.021s sys 0.033s
I20260317 16:16:38.913223 17827 leader_election.cc:290] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:38.912305 17808 tablet_bootstrap.cc:492] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.913885 17808 ts_tablet_manager.cc:1403] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.018s
I20260317 16:16:38.914729 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:38.915547 17767 raft_consensus.cc:3060] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.918354 17807 raft_consensus.cc:359] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.919246 17807 raft_consensus.cc:385] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.919432 17809 tablet_bootstrap.cc:492] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e: No bootstrap required, opened a new log
I20260317 16:16:38.919838 17807 raft_consensus.cc:740] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.920029 17809 ts_tablet_manager.cc:1403] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.000s
I20260317 16:16:38.920872 17807 consensus_queue.cc:260] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.922688 17809 raft_consensus.cc:359] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.923384 17809 raft_consensus.cc:385] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.923452 17693 raft_consensus.cc:2468] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 1.
I20260317 16:16:38.923689 17809 raft_consensus.cc:740] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.924338 17809 consensus_queue.cc:260] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.918962 17808 raft_consensus.cc:359] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.925235 17808 raft_consensus.cc:385] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.925505 17808 raft_consensus.cc:740] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.925699 17807 ts_tablet_manager.cc:1434] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20260317 16:16:38.926040 17767 raft_consensus.cc:2468] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 1.
I20260317 16:16:38.926131 17808 consensus_queue.cc:260] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.926745 17807 tablet_bootstrap.cc:492] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8: Bootstrap starting.
I20260317 16:16:38.927322 17582 leader_election.cc:304] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:38.928251 17808 ts_tablet_manager.cc:1434] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20260317 16:16:38.930137 17809 ts_tablet_manager.cc:1434] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260317 16:16:38.931427 17827 raft_consensus.cc:2804] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:38.931957 17827 raft_consensus.cc:697] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:38.933296 17808 tablet_bootstrap.cc:492] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.932821 17827 consensus_queue.cc:237] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.933916 17807 tablet_bootstrap.cc:654] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.940052 17808 tablet_bootstrap.cc:654] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.954910 17808 tablet_bootstrap.cc:492] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.955041 17807 tablet_bootstrap.cc:492] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8: No bootstrap required, opened a new log
I20260317 16:16:38.955384 17808 ts_tablet_manager.cc:1403] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.003s
I20260317 16:16:38.955416 17807 ts_tablet_manager.cc:1403] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.019s
I20260317 16:16:38.955585 17512 catalog_manager.cc:5671] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:38.958223 17807 raft_consensus.cc:359] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.958184 17808 raft_consensus.cc:359] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.959028 17807 raft_consensus.cc:385] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.959075 17808 raft_consensus.cc:385] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.959424 17807 raft_consensus.cc:740] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.959535 17808 raft_consensus.cc:740] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.960083 17807 consensus_queue.cc:260] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.960250 17808 consensus_queue.cc:260] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.962471 17808 ts_tablet_manager.cc:1434] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:16:38.962487 17807 ts_tablet_manager.cc:1434] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260317 16:16:38.963481 17808 tablet_bootstrap.cc:492] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: Bootstrap starting.
I20260317 16:16:38.969424 17812 raft_consensus.cc:493] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.969945 17808 tablet_bootstrap.cc:654] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: Neither blocks nor log segments found. Creating new log.
I20260317 16:16:38.969974 17812 raft_consensus.cc:515] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.972247 17812 leader_election.cc:290] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.973759 17812 raft_consensus.cc:493] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:38.974210 17812 raft_consensus.cc:515] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.974530 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:38.975045 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.975171 17618 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 0.
I20260317 16:16:38.976019 17808 tablet_bootstrap.cc:492] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: No bootstrap required, opened a new log
I20260317 16:16:38.976336 17812 leader_election.cc:290] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.976531 17808 ts_tablet_manager.cc:1403] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260317 16:16:38.976687 17733 leader_election.cc:304] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:38.977425 17812 raft_consensus.cc:2804] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.977800 17812 raft_consensus.cc:493] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:16:38.978193 17732 leader_election.cc:343] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:16:38.978351 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e38f8a85a03e423a8924458110a6cbf0" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:38.978546 17812 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.979038 17693 raft_consensus.cc:2468] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 0.
I20260317 16:16:38.979763 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e38f8a85a03e423a8924458110a6cbf0" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:38.980167 17732 leader_election.cc:304] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:38.980407 17618 raft_consensus.cc:2468] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 0.
I20260317 16:16:38.980924 17842 raft_consensus.cc:2804] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:38.981290 17842 raft_consensus.cc:493] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:38.981261 17808 raft_consensus.cc:359] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.981774 17842 raft_consensus.cc:3060] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.982096 17808 raft_consensus.cc:385] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:16:38.982437 17808 raft_consensus.cc:740] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Initialized, Role: FOLLOWER
I20260317 16:16:38.983242 17808 consensus_queue.cc:260] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.985587 17808 ts_tablet_manager.cc:1434] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260317 16:16:38.988412 17812 raft_consensus.cc:515] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.988691 17842 raft_consensus.cc:515] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:38.992972 17812 leader_election.cc:290] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:38.995177 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e38f8a85a03e423a8924458110a6cbf0" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:38.995985 17618 raft_consensus.cc:3060] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.996393 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:38.997006 17617 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:38.997251 17842 leader_election.cc:290] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:39.000224 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e38f8a85a03e423a8924458110a6cbf0" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:39.000844 17691 raft_consensus.cc:3060] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.004549 17618 raft_consensus.cc:2468] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:39.004873 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:39.005543 17693 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.005868 17617 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:39.005800 17733 leader_election.cc:304] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:39.006848 17842 raft_consensus.cc:2804] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:39.007495 17733 leader_election.cc:304] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:39.007591 17842 raft_consensus.cc:697] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:39.008469 17812 raft_consensus.cc:2804] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:39.008971 17812 raft_consensus.cc:697] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:39.008816 17842 consensus_queue.cc:237] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.010684 17691 raft_consensus.cc:2468] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:39.009865 17812 consensus_queue.cc:237] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.013281 17693 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 1.
I20260317 16:16:39.021631 17512 catalog_manager.cc:5671] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:39.037075 17512 catalog_manager.cc:5671] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:39.099376 17812 raft_consensus.cc:493] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:39.099937 17812 raft_consensus.cc:515] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.101547 17812 leader_election.cc:290] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:39.102574 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:39.102574 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e13d9b08e141a58988aa72744d54be" is_pre_election: true
I20260317 16:16:39.103597 17693 raft_consensus.cc:2393] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in current term 1: Already voted for candidate 9cb1383296ba48d6a985080124b4ad1e in this term.
I20260317 16:16:39.104401 17732 leader_election.cc:304] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c59c79684afa4b0c8dbf8ec87c7b07f8; no voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:39.105057 17812 raft_consensus.cc:3060] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.109701 17812 raft_consensus.cc:2749] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:39.117856 17814 raft_consensus.cc:493] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:39.118602 17814 raft_consensus.cc:515] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.120319 17814 leader_election.cc:290] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.121444 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38527a3689314a1aa8e83ff19e123d60" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:39.121351 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38527a3689314a1aa8e83ff19e123d60" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:39.121992 17767 raft_consensus.cc:2468] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 0.
I20260317 16:16:39.122010 17617 raft_consensus.cc:2468] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 0.
I20260317 16:16:39.123337 17659 leader_election.cc:304] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:39.124023 17814 raft_consensus.cc:2804] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:16:39.124328 17814 raft_consensus.cc:493] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:16:39.124595 17814 raft_consensus.cc:3060] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.129894 17814 raft_consensus.cc:515] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.131619 17814 leader_election.cc:290] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.132949 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38527a3689314a1aa8e83ff19e123d60" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:39.132858 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38527a3689314a1aa8e83ff19e123d60" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:39.134001 17767 raft_consensus.cc:3060] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.134001 17617 raft_consensus.cc:3060] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.138883 17617 raft_consensus.cc:2468] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 1.
I20260317 16:16:39.138820 17767 raft_consensus.cc:2468] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 1.
I20260317 16:16:39.141021 17659 leader_election.cc:304] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:39.142012 17814 raft_consensus.cc:2804] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:16:39.143178 17814 raft_consensus.cc:697] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 1 LEADER]: Becoming Leader. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:39.143713 17812 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.144037 17814 consensus_queue.cc:237] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.153201 17512 catalog_manager.cc:5671] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be reported cstate change: term changed from 0 to 1, leader changed from <none> to 06e13d9b08e141a58988aa72744d54be (127.16.123.194). New cstate: current_term: 1 leader_uuid: "06e13d9b08e141a58988aa72744d54be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:39.159286 17693 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.168951 17618 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.171607 17832 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.172565 17842 consensus_queue.cc:1048] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:39.181511 17842 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.204326 17843 consensus_queue.cc:1048] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.209259 17812 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.213393 17846 consensus_queue.cc:1048] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.214191 17832 consensus_queue.cc:1048] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:39.235543 17846 consensus_queue.cc:1048] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.266700 17846 consensus_queue.cc:1048] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:39.288309 17871 raft_consensus.cc:493] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:16:39.288756 17871 raft_consensus.cc:515] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.292627 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" is_pre_election: true
I20260317 16:16:39.293849 17767 raft_consensus.cc:2393] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06e13d9b08e141a58988aa72744d54be in current term 1: Already voted for candidate 9cb1383296ba48d6a985080124b4ad1e in this term.
I20260317 16:16:39.293637 17616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1383296ba48d6a985080124b4ad1e" is_pre_election: true
I20260317 16:16:39.294143 17871 leader_election.cc:290] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.296558 17657 leader_election.cc:304] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be; no voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:39.297272 17871 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:16:39.304271 17871 raft_consensus.cc:2749] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:16:39.314661 17767 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
mode: GRACEFUL
new_leader_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:43082
I20260317 16:16:39.315433 17767 raft_consensus.cc:606] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Received request to transfer leadership to TS 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:39.325420 17767 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
mode: GRACEFUL
new_leader_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:43082
I20260317 16:16:39.326009 17767 raft_consensus.cc:606] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Received request to transfer leadership to TS 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:39.333571 17767 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
mode: GRACEFUL
new_leader_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:43082
I20260317 16:16:39.334663 17767 raft_consensus.cc:606] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Received request to transfer leadership to TS 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:39.342276 17767 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
mode: GRACEFUL
new_leader_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:43082
I20260317 16:16:39.342881 17767 raft_consensus.cc:606] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Received request to transfer leadership to TS 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:39.370201 17846 consensus_queue.cc:1048] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.375943 17846 consensus_queue.cc:1048] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.394126 17827 consensus_queue.cc:1048] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.395972 17832 consensus_queue.cc:1048] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.398531 17827 consensus_queue.cc:1048] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:39.418908 17842 raft_consensus.cc:993] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: : Instructing follower 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:39.419924 17832 raft_consensus.cc:1081] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Signalling peer 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:39.424131 17691 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:50984
I20260317 16:16:39.424785 17691 raft_consensus.cc:493] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:39.425186 17691 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.433511 17691 raft_consensus.cc:515] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.433772 17832 consensus_queue.cc:1048] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:39.438130 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:39.438813 17767 raft_consensus.cc:3055] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:39.439137 17767 raft_consensus.cc:740] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:39.440021 17691 leader_election.cc:290] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.440002 17767 consensus_queue.cc:260] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.440927 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:39.441139 17767 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.441547 17617 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.448900 17617 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:39.449404 17767 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:39.450603 17659 leader_election.cc:304] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:39.451959 17871 raft_consensus.cc:2804] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:39.453258 17871 raft_consensus.cc:697] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Leader. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:39.456584 17871 consensus_queue.cc:237] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.459873 17832 consensus_queue.cc:1048] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.466496 17516 catalog_manager.cc:5671] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be reported cstate change: term changed from 1 to 2, leader changed from c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195) to 06e13d9b08e141a58988aa72744d54be (127.16.123.194). New cstate: current_term: 2 leader_uuid: "06e13d9b08e141a58988aa72744d54be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:39.611227 17842 raft_consensus.cc:993] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: : Instructing follower 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:39.611680 17842 raft_consensus.cc:1081] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Signalling peer 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:39.613523 17616 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:48382
I20260317 16:16:39.614123 17616 raft_consensus.cc:493] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:39.614504 17616 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.616479 17871 consensus_queue.cc:1048] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:16:39.623909 17616 raft_consensus.cc:515] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.626878 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:39.627116 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:39.627501 17691 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.627727 17767 raft_consensus.cc:3055] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:39.628095 17767 raft_consensus.cc:740] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:39.628897 17767 consensus_queue.cc:260] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.630230 17767 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.636323 17871 consensus_queue.cc:1048] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.647804 17767 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 2.
I20260317 16:16:39.636713 17616 leader_election.cc:290] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 2 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.651271 17582 leader_election.cc:304] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:39.650520 17691 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 2.
I20260317 16:16:39.656737 17843 raft_consensus.cc:2804] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:39.657923 17843 raft_consensus.cc:697] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:39.659929 17843 consensus_queue.cc:237] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.671623 17893 raft_consensus.cc:993] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: : Instructing follower 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:39.672745 17842 raft_consensus.cc:1081] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Signalling peer 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:39.670933 17516 catalog_manager.cc:5671] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 1 to 2, leader changed from c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195) to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 2 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:39.675710 17693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:50984
I20260317 16:16:39.677567 17693 raft_consensus.cc:493] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:39.678094 17693 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.686918 17693 raft_consensus.cc:515] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.688611 17693 leader_election.cc:290] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.689709 17765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:39.689702 17616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:39.690407 17765 raft_consensus.cc:3055] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:39.690966 17765 raft_consensus.cc:740] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:39.691039 17616 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.693118 17765 consensus_queue.cc:260] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.694682 17765 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.701584 17616 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:39.701843 17765 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:39.703410 17657 leader_election.cc:304] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:39.704288 17897 raft_consensus.cc:2804] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:39.705003 17897 raft_consensus.cc:697] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Leader. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:39.705960 17897 consensus_queue.cc:237] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.713514 17516 catalog_manager.cc:5671] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be reported cstate change: term changed from 1 to 2, leader changed from c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195) to 06e13d9b08e141a58988aa72744d54be (127.16.123.194). New cstate: current_term: 2 leader_uuid: "06e13d9b08e141a58988aa72744d54be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:39.828773 17832 raft_consensus.cc:993] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: : Instructing follower 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:39.829561 17893 raft_consensus.cc:1081] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Signalling peer 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:39.831219 17693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:50984
I20260317 16:16:39.831830 17693 raft_consensus.cc:493] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:39.832314 17693 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.838198 17693 raft_consensus.cc:515] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.841214 17693 leader_election.cc:290] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:39.841193 17616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:39.841770 17765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:39.841894 17616 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.842377 17765 raft_consensus.cc:3055] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:39.842734 17765 raft_consensus.cc:740] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:39.843520 17765 consensus_queue.cc:260] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.844702 17765 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:39.847407 17616 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:39.848511 17659 leader_election.cc:304] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:39.849325 17897 raft_consensus.cc:2804] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:39.849750 17897 raft_consensus.cc:697] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Leader. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:39.850554 17765 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:39.850569 17897 consensus_queue.cc:237] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:39.859309 17516 catalog_manager.cc:5671] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be reported cstate change: term changed from 1 to 2, leader changed from c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195) to 06e13d9b08e141a58988aa72744d54be (127.16.123.194). New cstate: current_term: 2 leader_uuid: "06e13d9b08e141a58988aa72744d54be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:39.917598 17616 raft_consensus.cc:1275] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:39.918902 17897 consensus_queue.cc:1048] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:16:39.931520 17765 raft_consensus.cc:1275] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:39.932816 17814 consensus_queue.cc:1048] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:40.159765 17765 raft_consensus.cc:1275] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:40.161092 17843 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:40.168787 17693 raft_consensus.cc:1275] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:40.170403 17900 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:40.191742 17616 raft_consensus.cc:1275] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:40.193065 17871 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:40.203178 17765 raft_consensus.cc:1275] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:40.205257 17897 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:40.279527 17765 raft_consensus.cc:1275] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:40.280730 17897 consensus_queue.cc:1048] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:40.289959 17616 raft_consensus.cc:1275] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:40.291730 17814 consensus_queue.cc:1048] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:42.383701 16879 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260317 16:16:42.384497 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:42.411603 17616 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
mode: GRACEFUL
new_leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:48412
I20260317 16:16:42.412221 17616 raft_consensus.cc:606] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 LEADER]: Received request to transfer leadership to TS c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:42.414595 17616 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
mode: GRACEFUL
new_leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:48412
I20260317 16:16:42.415107 17616 raft_consensus.cc:606] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Received request to transfer leadership to TS c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:42.416059 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260317 16:16:42.416455 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:42.674877 17908 raft_consensus.cc:993] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: : Instructing follower c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:42.675673 17905 raft_consensus.cc:1081] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 LEADER]: Signalling peer c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:42.677374 17767 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:43054
I20260317 16:16:42.678082 17767 raft_consensus.cc:493] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:42.678422 17767 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:42.684321 17767 raft_consensus.cc:515] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:42.686169 17767 leader_election.cc:290] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 3 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:42.687335 17616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:42.687438 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:42.688198 17616 raft_consensus.cc:3055] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:42.688205 17691 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:42.688706 17616 raft_consensus.cc:740] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:42.689651 17616 consensus_queue.cc:260] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:42.691224 17616 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:42.697269 17691 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 3.
I20260317 16:16:42.699235 17732 leader_election.cc:304] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:42.699460 17616 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 3.
I20260317 16:16:42.700294 17907 raft_consensus.cc:2804] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:42.701377 17907 raft_consensus.cc:697] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:42.702328 17907 consensus_queue.cc:237] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:42.711401 17512 catalog_manager.cc:5671] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 2 to 3, leader changed from 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193) to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:42.755851 17908 raft_consensus.cc:993] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: : Instructing follower c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:42.756309 17905 raft_consensus.cc:1081] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Signalling peer c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:42.757789 17767 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:43054
I20260317 16:16:42.758376 17767 raft_consensus.cc:493] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:42.758670 17767 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:42.764358 17767 raft_consensus.cc:515] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:42.766685 17767 leader_election.cc:290] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:42.767400 17616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:42.768049 17616 raft_consensus.cc:3055] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:42.768349 17616 raft_consensus.cc:740] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:42.768384 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:42.769121 17691 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:42.769398 17616 consensus_queue.cc:260] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:42.771147 17616 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:42.776638 17691 raft_consensus.cc:2468] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 2.
I20260317 16:16:42.778087 17616 raft_consensus.cc:2468] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 2.
I20260317 16:16:42.778290 17732 leader_election.cc:304] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:42.779109 17907 raft_consensus.cc:2804] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:42.779536 17907 raft_consensus.cc:697] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:42.780340 17907 consensus_queue.cc:237] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:42.791270 17512 catalog_manager.cc:5671] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 1 to 2, leader changed from 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193) to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:43.166152 17616 raft_consensus.cc:1275] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:43.167533 17907 consensus_queue.cc:1048] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:43.179206 17691 raft_consensus.cc:1275] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:43.182744 17911 consensus_queue.cc:1048] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:43.251866 17691 raft_consensus.cc:1275] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:43.253181 17911 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:43.263996 17616 raft_consensus.cc:1275] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:43.265341 17920 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:43.417306 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:43.422856 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:43.423290 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:44.424346 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:44.432242 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:44.432746 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:45.433941 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:45.441416 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:45.441929 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:46.443030 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:46.448906 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:46.449342 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:47.450423 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:47.460026 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:47.463963 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:48.467057 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:48.474457 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:48.475122 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:49.476212 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:49.481657 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:49.482121 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:50.483165 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:50.488838 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:50.489287 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:51.490545 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:51.498487 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:51.498961 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:52.536334 16879 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260317 16:16:52.543542 17767 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
mode: GRACEFUL
new_leader_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:43082
I20260317 16:16:52.544211 17767 raft_consensus.cc:606] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Received request to transfer leadership to TS 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:52.552243 17767 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
mode: GRACEFUL
new_leader_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:43082
I20260317 16:16:52.552665 17767 raft_consensus.cc:606] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 LEADER]: Received request to transfer leadership to TS 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:52.579754 17691 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
mode: GRACEFUL
new_leader_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:35922
I20260317 16:16:52.580574 17691 raft_consensus.cc:606] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Received request to transfer leadership to TS 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:52.586894 17691 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
mode: GRACEFUL
new_leader_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:35922
I20260317 16:16:52.587610 17691 raft_consensus.cc:606] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Received request to transfer leadership to TS 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:52.593950 17691 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
mode: GRACEFUL
new_leader_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:35922
I20260317 16:16:52.594573 17691 raft_consensus.cc:606] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Received request to transfer leadership to TS 9cb1383296ba48d6a985080124b4ad1e
I20260317 16:16:52.684260 17897 raft_consensus.cc:993] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: : Instructing follower 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.684707 17897 raft_consensus.cc:1081] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Signalling peer 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.686447 17616 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:48396
I20260317 16:16:52.686955 17616 raft_consensus.cc:493] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:52.687300 17616 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.693312 17616 raft_consensus.cc:515] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.695051 17616 leader_election.cc:290] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:52.696200 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:52.696177 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c2c3cb2259449ab45cc2db20e46ef6" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:52.697244 17767 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.697244 17691 raft_consensus.cc:3055] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:52.697772 17691 raft_consensus.cc:740] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:52.698686 17691 consensus_queue.cc:260] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.700094 17691 raft_consensus.cc:3060] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.704859 17767 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:52.706489 17582 leader_election.cc:304] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:52.706913 17691 raft_consensus.cc:2468] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:52.707746 17926 raft_consensus.cc:2804] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:52.708835 17926 raft_consensus.cc:697] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:52.709909 17926 consensus_queue.cc:237] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.719352 17512 catalog_manager.cc:5671] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 2 to 3, leader changed from 06e13d9b08e141a58988aa72744d54be (127.16.123.194) to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 3 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:52.868706 17920 raft_consensus.cc:993] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: : Instructing follower 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.869386 17940 raft_consensus.cc:1081] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 LEADER]: Signalling peer 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.871317 17616 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:48382
I20260317 16:16:52.871853 17616 raft_consensus.cc:493] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:52.872227 17616 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:52.875962 17933 raft_consensus.cc:993] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: : Instructing follower 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.876484 17897 raft_consensus.cc:1081] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Signalling peer 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.878347 17617 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:48396
I20260317 16:16:52.879061 17617 raft_consensus.cc:493] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:52.878949 17616 raft_consensus.cc:515] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.879442 17617 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.880520 17616 leader_election.cc:290] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 4 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:52.881366 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:52.881502 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f286f6866fc4d37a61f21afe7b892fe" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:52.881899 17691 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:52.881924 17767 raft_consensus.cc:3055] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:52.882355 17767 raft_consensus.cc:740] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:52.883217 17767 consensus_queue.cc:260] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.884516 17767 raft_consensus.cc:3060] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:52.887050 17617 raft_consensus.cc:515] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.888825 17617 leader_election.cc:290] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:52.889971 17765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:52.889915 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:52.888780 17691 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 4.
I20260317 16:16:52.890610 17765 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.890692 17693 raft_consensus.cc:3055] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:52.890954 17767 raft_consensus.cc:2468] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 4.
I20260317 16:16:52.891153 17693 raft_consensus.cc:740] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:52.892300 17582 leader_election.cc:304] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:52.892032 17693 consensus_queue.cc:260] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.893235 17926 raft_consensus.cc:2804] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:52.893575 17693 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.893738 17926 raft_consensus.cc:697] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 4 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:52.895179 17926 consensus_queue.cc:237] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.897681 17765 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:52.898568 17582 leader_election.cc:304] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:52.899464 17926 raft_consensus.cc:2804] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:52.900267 17926 raft_consensus.cc:697] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:52.901204 17926 consensus_queue.cc:237] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.903966 17693 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:52.903837 17512 catalog_manager.cc:5671] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 3 to 4, leader changed from c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195) to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 4 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:52.916342 17512 catalog_manager.cc:5671] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 2 to 3, leader changed from 06e13d9b08e141a58988aa72744d54be (127.16.123.194) to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 3 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:52.938344 17920 raft_consensus.cc:993] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: : Instructing follower 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.938859 17920 raft_consensus.cc:1081] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Signalling peer 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:52.940699 17617 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:48382
I20260317 16:16:52.941237 17617 raft_consensus.cc:493] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:52.941505 17617 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.948752 17617 raft_consensus.cc:515] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.950502 17617 leader_election.cc:290] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:52.951530 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:52.951543 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abad741d31214ee4b0e8b8a3bbb2ead7" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:52.952142 17767 raft_consensus.cc:3055] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:52.952544 17693 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.952680 17767 raft_consensus.cc:740] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:52.953469 17767 consensus_queue.cc:260] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.954424 17767 raft_consensus.cc:3060] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:52.957448 17693 raft_consensus.cc:2468] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:52.958618 17583 leader_election.cc:304] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, 9cb1383296ba48d6a985080124b4ad1e; no voters:
I20260317 16:16:52.959530 17926 raft_consensus.cc:2804] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:52.959571 17767 raft_consensus.cc:2468] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:52.960286 17926 raft_consensus.cc:697] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:52.961481 17926 consensus_queue.cc:237] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:52.970052 17512 catalog_manager.cc:5671] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 2 to 3, leader changed from c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195) to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 3 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:53.013314 17933 raft_consensus.cc:993] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: : Instructing follower 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:53.013758 17933 raft_consensus.cc:1081] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Signalling peer 9cb1383296ba48d6a985080124b4ad1e to start an election
I20260317 16:16:53.015512 17617 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
from {username='slave'} at 127.0.0.1:48396
I20260317 16:16:53.016224 17617 raft_consensus.cc:493] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:53.016553 17617 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:53.021621 17617 raft_consensus.cc:515] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:53.023312 17617 leader_election.cc:290] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:53.024340 17693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:53.024350 17767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a4694aee514c9e8f65ebab0a511dc3" candidate_uuid: "9cb1383296ba48d6a985080124b4ad1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:53.024895 17693 raft_consensus.cc:3055] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:16:53.025174 17767 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:53.025197 17693 raft_consensus.cc:740] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:53.026079 17693 consensus_queue.cc:260] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:53.027103 17693 raft_consensus.cc:3060] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:16:53.030388 17767 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:53.031448 17693 raft_consensus.cc:2468] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1383296ba48d6a985080124b4ad1e in term 3.
I20260317 16:16:53.031425 17582 leader_election.cc:304] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cb1383296ba48d6a985080124b4ad1e, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:53.032675 17926 raft_consensus.cc:2804] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:16:53.033080 17926 raft_consensus.cc:697] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Becoming Leader. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:53.033787 17926 consensus_queue.cc:237] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:53.040973 17512 catalog_manager.cc:5671] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e reported cstate change: term changed from 2 to 3, leader changed from 06e13d9b08e141a58988aa72744d54be (127.16.123.194) to 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193). New cstate: current_term: 3 leader_uuid: "9cb1383296ba48d6a985080124b4ad1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:53.161820 17693 raft_consensus.cc:1275] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.163630 17926 consensus_queue.cc:1048] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:53.174827 17767 raft_consensus.cc:1275] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.176656 17934 consensus_queue.cc:1048] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:53.379593 17767 raft_consensus.cc:1275] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:53.380884 17934 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:16:53.381664 17767 raft_consensus.cc:1275] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.382967 17934 consensus_queue.cc:1048] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:53.399294 17693 raft_consensus.cc:1275] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:53.402573 17953 consensus_queue.cc:1048] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:16:53.404112 17693 raft_consensus.cc:1275] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.409686 17767 raft_consensus.cc:1275] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.410895 17947 consensus_queue.cc:1048] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:53.414881 17954 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:53.435360 17691 raft_consensus.cc:1275] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.437284 17956 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:53.498070 17764 raft_consensus.cc:1275] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.499738 17956 consensus_queue.cc:1048] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:16:53.509641 17693 raft_consensus.cc:1275] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Refusing update from remote peer 9cb1383296ba48d6a985080124b4ad1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:16:53.511081 17955 consensus_queue.cc:1048] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:16:55.665202 16879 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260317 16:16:55.665975 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:55.678591 17617 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
mode: GRACEFUL
new_leader_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:48412
I20260317 16:16:55.679320 17617 raft_consensus.cc:606] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Received request to transfer leadership to TS 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:55.684621 17617 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
mode: GRACEFUL
new_leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:48412
I20260317 16:16:55.685281 17617 raft_consensus.cc:606] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Received request to transfer leadership to TS c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:55.689323 17617 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
mode: GRACEFUL
new_leader_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:48412
I20260317 16:16:55.689862 17617 raft_consensus.cc:606] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Received request to transfer leadership to TS 06e13d9b08e141a58988aa72744d54be
I20260317 16:16:55.692759 17617 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cbad27cc136042689fa5425accbb4419"
dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
mode: GRACEFUL
new_leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:48412
I20260317 16:16:55.693975 17617 raft_consensus.cc:606] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Received request to transfer leadership to TS c59c79684afa4b0c8dbf8ec87c7b07f8
I20260317 16:16:55.695833 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260317 16:16:55.696578 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:56.034241 17964 raft_consensus.cc:993] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: : Instructing follower 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:56.035203 17955 raft_consensus.cc:1081] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Signalling peer 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:56.037266 17691 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:50996
I20260317 16:16:56.037707 17691 raft_consensus.cc:493] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:56.037956 17691 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:56.042919 17691 raft_consensus.cc:515] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.045303 17691 leader_election.cc:290] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 4 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:56.046607 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:56.046730 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b1a38ac52d14630bd9aec18edd53793" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:56.047286 17764 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:56.047495 17617 raft_consensus.cc:3055] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:16:56.047959 17617 raft_consensus.cc:740] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:56.048754 17617 consensus_queue.cc:260] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.049928 17617 raft_consensus.cc:3060] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:16:56.053879 17764 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 4.
I20260317 16:16:56.055801 17657 leader_election.cc:304] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:56.057070 17963 raft_consensus.cc:2804] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:16:56.057245 17617 raft_consensus.cc:2468] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 4.
I20260317 16:16:56.058517 17963 raft_consensus.cc:697] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 4 LEADER]: Becoming Leader. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:56.059514 17963 consensus_queue.cc:237] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.070329 17512 catalog_manager.cc:5671] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be reported cstate change: term changed from 3 to 4, leader changed from 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193) to 06e13d9b08e141a58988aa72744d54be (127.16.123.194). New cstate: current_term: 4 leader_uuid: "06e13d9b08e141a58988aa72744d54be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:56.085916 17964 raft_consensus.cc:993] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: : Instructing follower 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:56.086460 17955 raft_consensus.cc:1081] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Signalling peer 06e13d9b08e141a58988aa72744d54be to start an election
I20260317 16:16:56.088891 17691 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d"
dest_uuid: "06e13d9b08e141a58988aa72744d54be"
from {username='slave'} at 127.0.0.1:50996
I20260317 16:16:56.089417 17691 raft_consensus.cc:493] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:56.089680 17691 raft_consensus.cc:3060] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.095244 17691 raft_consensus.cc:515] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.097232 17691 leader_election.cc:290] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195:33255)
I20260317 16:16:56.098557 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
I20260317 16:16:56.098557 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d1d1ab50904f4a936aaefffcdfd42d" candidate_uuid: "06e13d9b08e141a58988aa72744d54be" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:56.099592 17764 raft_consensus.cc:3060] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.099601 17617 raft_consensus.cc:3055] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:56.100209 17617 raft_consensus.cc:740] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:56.101066 17617 consensus_queue.cc:260] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.102411 17617 raft_consensus.cc:3060] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.108850 17764 raft_consensus.cc:2468] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:56.110415 17617 raft_consensus.cc:2468] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06e13d9b08e141a58988aa72744d54be in term 2.
I20260317 16:16:56.110491 17657 leader_election.cc:304] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:56.111536 17963 raft_consensus.cc:2804] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:56.112234 17963 raft_consensus.cc:697] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Becoming Leader. State: Replica: 06e13d9b08e141a58988aa72744d54be, State: Running, Role: LEADER
I20260317 16:16:56.113336 17963 consensus_queue.cc:237] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.122655 17512 catalog_manager.cc:5671] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be reported cstate change: term changed from 1 to 2, leader changed from 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193) to 06e13d9b08e141a58988aa72744d54be (127.16.123.194). New cstate: current_term: 2 leader_uuid: "06e13d9b08e141a58988aa72744d54be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260317 16:16:56.125212 17964 raft_consensus.cc:993] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: : Instructing follower c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:56.125787 17955 raft_consensus.cc:1081] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Signalling peer c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:56.127840 17764 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cbad27cc136042689fa5425accbb4419"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:43054
I20260317 16:16:56.128391 17764 raft_consensus.cc:493] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:56.128831 17764 raft_consensus.cc:3060] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.140295 17764 raft_consensus.cc:515] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.143887 17764 leader_election.cc:290] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 2 election: Requested vote from peers 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955), 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607)
I20260317 16:16:56.145349 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:56.145661 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbad27cc136042689fa5425accbb4419" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:56.146433 17691 raft_consensus.cc:3060] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.146610 17617 raft_consensus.cc:3055] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:56.147337 17617 raft_consensus.cc:740] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:56.148077 17617 consensus_queue.cc:260] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.149413 17617 raft_consensus.cc:3060] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.154892 17691 raft_consensus.cc:2468] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 2.
I20260317 16:16:56.156592 17732 leader_election.cc:304] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:56.157683 17617 raft_consensus.cc:2468] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 2.
I20260317 16:16:56.158061 17969 raft_consensus.cc:2804] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:56.160115 17969 raft_consensus.cc:697] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:56.161851 17969 consensus_queue.cc:237] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.171561 17512 catalog_manager.cc:5671] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 1 to 2, leader changed from 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193) to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:56.206216 17964 raft_consensus.cc:993] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: : Instructing follower c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:56.207233 17955 raft_consensus.cc:1081] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Signalling peer c59c79684afa4b0c8dbf8ec87c7b07f8 to start an election
I20260317 16:16:56.209468 17764 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15"
dest_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8"
from {username='slave'} at 127.0.0.1:43054
I20260317 16:16:56.210454 17764 raft_consensus.cc:493] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:16:56.210989 17764 raft_consensus.cc:3060] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.220070 17764 raft_consensus.cc:515] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.222739 17764 leader_election.cc:290] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607), 06e13d9b08e141a58988aa72744d54be (127.16.123.194:32955)
I20260317 16:16:56.223582 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9cb1383296ba48d6a985080124b4ad1e"
I20260317 16:16:56.224027 17691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ecf40a1e6da4628b84dbfb94499ff15" candidate_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06e13d9b08e141a58988aa72744d54be"
I20260317 16:16:56.224339 17617 raft_consensus.cc:3055] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:16:56.224746 17691 raft_consensus.cc:3060] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.224759 17617 raft_consensus.cc:740] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cb1383296ba48d6a985080124b4ad1e, State: Running, Role: LEADER
I20260317 16:16:56.232481 17617 consensus_queue.cc:260] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.233744 17617 raft_consensus.cc:3060] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:16:56.231240 17691 raft_consensus.cc:2468] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 2.
I20260317 16:16:56.242256 17617 raft_consensus.cc:2468] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59c79684afa4b0c8dbf8ec87c7b07f8 in term 2.
I20260317 16:16:56.242261 17732 leader_election.cc:304] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e13d9b08e141a58988aa72744d54be, c59c79684afa4b0c8dbf8ec87c7b07f8; no voters:
I20260317 16:16:56.243492 17969 raft_consensus.cc:2804] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:16:56.243904 17969 raft_consensus.cc:697] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Becoming Leader. State: Replica: c59c79684afa4b0c8dbf8ec87c7b07f8, State: Running, Role: LEADER
I20260317 16:16:56.244858 17969 consensus_queue.cc:237] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } }
I20260317 16:16:56.253507 17512 catalog_manager.cc:5671] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 reported cstate change: term changed from 1 to 2, leader changed from 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193) to c59c79684afa4b0c8dbf8ec87c7b07f8 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 } health_report { overall_health: HEALTHY } } }
I20260317 16:16:56.537534 17617 raft_consensus.cc:1275] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:56.539165 17963 consensus_queue.cc:1048] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:56.546886 17616 raft_consensus.cc:1275] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:56.549348 17963 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:16:56.553889 17764 raft_consensus.cc:1275] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:56.557121 17978 consensus_queue.cc:1048] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:56.574651 17767 raft_consensus.cc:1275] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Refusing update from remote peer 06e13d9b08e141a58988aa72744d54be: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:16:56.578445 17963 consensus_queue.cc:1048] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59c79684afa4b0c8dbf8ec87c7b07f8" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:16:56.637512 17617 raft_consensus.cc:1275] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:56.639390 17969 consensus_queue.cc:1048] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:56.653331 17691 raft_consensus.cc:1275] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:56.658171 17973 consensus_queue.cc:1048] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:56.697679 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:56.708666 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:56.709549 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:56.754215 17691 raft_consensus.cc:1275] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:56.756435 17969 consensus_queue.cc:1048] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e13d9b08e141a58988aa72744d54be" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:16:56.773840 17617 raft_consensus.cc:1275] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Refusing update from remote peer c59c79684afa4b0c8dbf8ec87c7b07f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:16:56.776216 17988 consensus_queue.cc:1048] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1383296ba48d6a985080124b4ad1e" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:16:57.711164 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:57.717928 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:57.718488 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:58.719693 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:58.727380 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:58.727980 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:16:59.729007 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:16:59.736486 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:16:59.737007 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:00.738270 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:00.748456 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:17:00.749189 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:01.750470 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:01.757357 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:17:01.758361 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:02.759586 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:02.769402 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:17:02.770102 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:03.771514 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:03.779215 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:17:03.779903 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:04.781081 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:04.787598 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:17:04.788128 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:05.834548 16879 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260317 16:17:05.835208 16879 tablet_server.cc:178] TabletServer@127.16.123.193:0 shutting down...
I20260317 16:17:05.887063 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:17:05.887977 16879 tablet_replica.cc:333] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.889024 16879 raft_consensus.cc:2243] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:05.889956 16879 raft_consensus.cc:2272] T e38f8a85a03e423a8924458110a6cbf0 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.893234 16879 tablet_replica.cc:333] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.893959 16879 raft_consensus.cc:2243] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:05.894436 16879 raft_consensus.cc:2272] T 5ecf40a1e6da4628b84dbfb94499ff15 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.897150 16879 tablet_replica.cc:333] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.897966 16879 raft_consensus.cc:2243] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:05.899560 16879 raft_consensus.cc:2272] T abad741d31214ee4b0e8b8a3bbb2ead7 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.902434 16879 tablet_replica.cc:333] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.903162 16879 raft_consensus.cc:2243] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:05.903796 16879 raft_consensus.cc:2272] T 38527a3689314a1aa8e83ff19e123d60 P 9cb1383296ba48d6a985080124b4ad1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.906358 16879 tablet_replica.cc:333] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.907013 16879 raft_consensus.cc:2243] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:17:05.907866 16879 raft_consensus.cc:2272] T 8f286f6866fc4d37a61f21afe7b892fe P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.910535 16879 tablet_replica.cc:333] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.911147 16879 raft_consensus.cc:2243] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:05.912011 16879 raft_consensus.cc:2272] T d4c2c3cb2259449ab45cc2db20e46ef6 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.914415 16879 tablet_replica.cc:333] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.915490 16879 raft_consensus.cc:2243] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:05.916841 16879 raft_consensus.cc:2272] T 8b1a38ac52d14630bd9aec18edd53793 P 9cb1383296ba48d6a985080124b4ad1e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.919643 16879 tablet_replica.cc:333] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.920167 16879 raft_consensus.cc:2243] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:05.920570 16879 raft_consensus.cc:2272] T cbad27cc136042689fa5425accbb4419 P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.922513 16879 tablet_replica.cc:333] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
I20260317 16:17:05.923136 16879 raft_consensus.cc:2243] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:05.923586 16879 raft_consensus.cc:2272] T 24d1d1ab50904f4a936aaefffcdfd42d P 9cb1383296ba48d6a985080124b4ad1e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:05.925819 16879 tablet_replica.cc:333] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e: stopping tablet replica
W20260317 16:17:05.926713 17733 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
I20260317 16:17:05.927016 16879 raft_consensus.cc:2243] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:05.928912 16879 raft_consensus.cc:2272] T 07a4694aee514c9e8f65ebab0a511dc3 P 9cb1383296ba48d6a985080124b4ad1e [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:05.931988 17733 consensus_peers.cc:597] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 -> Peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Couldn't send request to peer 9cb1383296ba48d6a985080124b4ad1e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:05.961735 16879 tablet_server.cc:195] TabletServer@127.16.123.193:0 shutdown complete.
W20260317 16:17:05.970755 17659 consensus_peers.cc:597] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be -> Peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Couldn't send request to peer 9cb1383296ba48d6a985080124b4ad1e. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:35607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:05.985220 16879 tablet_server.cc:178] TabletServer@127.16.123.194:0 shutting down...
I20260317 16:17:06.019838 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:17:06.020818 16879 tablet_replica.cc:333] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.021986 16879 raft_consensus.cc:2243] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.022598 16879 raft_consensus.cc:2272] T e38f8a85a03e423a8924458110a6cbf0 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.024886 16879 tablet_replica.cc:333] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.025508 16879 raft_consensus.cc:2243] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.026433 16879 raft_consensus.cc:2272] T 5ecf40a1e6da4628b84dbfb94499ff15 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.028808 16879 tablet_replica.cc:333] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.029381 16879 raft_consensus.cc:2243] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.029870 16879 raft_consensus.cc:2272] T abad741d31214ee4b0e8b8a3bbb2ead7 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.031767 16879 tablet_replica.cc:333] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.032372 16879 raft_consensus.cc:2243] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.033206 16879 raft_consensus.cc:2272] T 38527a3689314a1aa8e83ff19e123d60 P 06e13d9b08e141a58988aa72744d54be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.035174 16879 tablet_replica.cc:333] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.035938 16879 raft_consensus.cc:2243] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.036419 16879 raft_consensus.cc:2272] T cbad27cc136042689fa5425accbb4419 P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.038105 16879 tablet_replica.cc:333] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.038580 16879 raft_consensus.cc:2243] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.039109 16879 raft_consensus.cc:2272] T 07a4694aee514c9e8f65ebab0a511dc3 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.041507 16879 tablet_replica.cc:333] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.042054 16879 raft_consensus.cc:2243] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.043100 16879 raft_consensus.cc:2272] T 24d1d1ab50904f4a936aaefffcdfd42d P 06e13d9b08e141a58988aa72744d54be [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.045171 16879 tablet_replica.cc:333] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.045671 16879 raft_consensus.cc:2243] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.046487 16879 raft_consensus.cc:2272] T 8b1a38ac52d14630bd9aec18edd53793 P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.048524 16879 tablet_replica.cc:333] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.049295 16879 raft_consensus.cc:2243] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.049765 16879 raft_consensus.cc:2272] T d4c2c3cb2259449ab45cc2db20e46ef6 P 06e13d9b08e141a58988aa72744d54be [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.052109 16879 tablet_replica.cc:333] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be: stopping tablet replica
I20260317 16:17:06.052645 16879 raft_consensus.cc:2243] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.053194 16879 raft_consensus.cc:2272] T 8f286f6866fc4d37a61f21afe7b892fe P 06e13d9b08e141a58988aa72744d54be [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.082237 16879 tablet_server.cc:195] TabletServer@127.16.123.194:0 shutdown complete.
I20260317 16:17:06.103824 16879 tablet_server.cc:178] TabletServer@127.16.123.195:0 shutting down...
W20260317 16:17:06.130623 17733 consensus_peers.cc:597] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 -> Peer 9cb1383296ba48d6a985080124b4ad1e (127.16.123.193:35607): Couldn't send request to peer 9cb1383296ba48d6a985080124b4ad1e. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:35607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:06.140206 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:17:06.141074 16879 tablet_replica.cc:333] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.141870 16879 raft_consensus.cc:2243] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.143219 16879 raft_consensus.cc:2272] T e38f8a85a03e423a8924458110a6cbf0 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.145052 16879 tablet_replica.cc:333] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.145747 16879 raft_consensus.cc:2243] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.147212 16879 raft_consensus.cc:2272] T 07a4694aee514c9e8f65ebab0a511dc3 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.149252 16879 tablet_replica.cc:333] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.149832 16879 raft_consensus.cc:2243] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.150312 16879 raft_consensus.cc:2272] T 24d1d1ab50904f4a936aaefffcdfd42d P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.152287 16879 tablet_replica.cc:333] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.152878 16879 raft_consensus.cc:2243] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.153335 16879 raft_consensus.cc:2272] T 38527a3689314a1aa8e83ff19e123d60 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.155323 16879 tablet_replica.cc:333] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.156185 16879 raft_consensus.cc:2243] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.156651 16879 raft_consensus.cc:2272] T abad741d31214ee4b0e8b8a3bbb2ead7 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.158519 16879 tablet_replica.cc:333] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.159205 16879 raft_consensus.cc:2243] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.160074 16879 raft_consensus.cc:2272] T 5ecf40a1e6da4628b84dbfb94499ff15 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.162120 16879 tablet_replica.cc:333] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.162657 16879 raft_consensus.cc:2243] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.163254 16879 raft_consensus.cc:2272] T 8f286f6866fc4d37a61f21afe7b892fe P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.165088 16879 tablet_replica.cc:333] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.165571 16879 raft_consensus.cc:2243] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.166015 16879 raft_consensus.cc:2272] T d4c2c3cb2259449ab45cc2db20e46ef6 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.167747 16879 tablet_replica.cc:333] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.168216 16879 raft_consensus.cc:2243] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:06.168653 16879 raft_consensus.cc:2272] T 8b1a38ac52d14630bd9aec18edd53793 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.170348 16879 tablet_replica.cc:333] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8: stopping tablet replica
I20260317 16:17:06.170853 16879 raft_consensus.cc:2243] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.171633 16879 raft_consensus.cc:2272] T cbad27cc136042689fa5425accbb4419 P c59c79684afa4b0c8dbf8ec87c7b07f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.201339 16879 tablet_server.cc:195] TabletServer@127.16.123.195:0 shutdown complete.
I20260317 16:17:06.222666 16879 master.cc:561] Master@127.16.123.254:40161 shutting down...
I20260317 16:17:06.253691 16879 raft_consensus.cc:2243] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:06.254441 16879 raft_consensus.cc:2272] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:06.254736 16879 tablet_replica.cc:333] T 00000000000000000000000000000000 P dc2066b9378844879b54182e6a8f8614: stopping tablet replica
I20260317 16:17:06.278551 16879 master.cc:583] Master@127.16.123.254:40161 shutdown complete.
06e13d9b08e141a58988aa72744d54be 4
9cb1383296ba48d6a985080124b4ad1e 5
c59c79684afa4b0c8dbf8ec87c7b07f8 1
06e13d9b08e141a58988aa72744d54be 4
9cb1383296ba48d6a985080124b4ad1e 3
c59c79684afa4b0c8dbf8ec87c7b07f8 3
06e13d9b08e141a58988aa72744d54be 1
9cb1383296ba48d6a985080124b4ad1e 8
c59c79684afa4b0c8dbf8ec87c7b07f8 1
06e13d9b08e141a58988aa72744d54be 3
9cb1383296ba48d6a985080124b4ad1e 4
c59c79684afa4b0c8dbf8ec87c7b07f8 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29133 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260317 16:17:06.333812 16879 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.123.254:39965
I20260317 16:17:06.335170 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:17:06.343287 18011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.343487 18010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.344575 18013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:17:06.345396 16879 server_base.cc:1060] running on GCE node
I20260317 16:17:06.346657 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:17:06.347832 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:17:06.348210 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764226348160 us; error 0 us; skew 500 ppm
I20260317 16:17:06.349058 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:17:06.352439 16879 webserver.cc:492] Webserver started at http://127.16.123.254:46515/ using document root <none> and password file <none>
I20260317 16:17:06.353088 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:17:06.353307 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:17:06.353641 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:17:06.355818 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/master-0-root/instance:
uuid: "387423d8ffe1441b90cc64abbbf247c7"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.360883 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260317 16:17:06.365022 18018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.366096 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260317 16:17:06.366552 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/master-0-root
uuid: "387423d8ffe1441b90cc64abbbf247c7"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.367127 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:17:06.387147 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:17:06.388592 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:17:06.441200 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.254:39965
I20260317 16:17:06.441329 18069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.254:39965 every 8 connection(s)
I20260317 16:17:06.446197 18070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:06.460673 18070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7: Bootstrap starting.
I20260317 16:17:06.466639 18070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:06.473116 18070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7: No bootstrap required, opened a new log
I20260317 16:17:06.475683 18070 raft_consensus.cc:359] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387423d8ffe1441b90cc64abbbf247c7" member_type: VOTER }
I20260317 16:17:06.476267 18070 raft_consensus.cc:385] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:06.476457 18070 raft_consensus.cc:740] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387423d8ffe1441b90cc64abbbf247c7, State: Initialized, Role: FOLLOWER
I20260317 16:17:06.476974 18070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387423d8ffe1441b90cc64abbbf247c7" member_type: VOTER }
I20260317 16:17:06.477463 18070 raft_consensus.cc:399] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 16:17:06.477659 18070 raft_consensus.cc:493] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 16:17:06.478322 18070 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:06.483979 18070 raft_consensus.cc:515] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387423d8ffe1441b90cc64abbbf247c7" member_type: VOTER }
I20260317 16:17:06.484522 18070 leader_election.cc:304] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 387423d8ffe1441b90cc64abbbf247c7; no voters:
I20260317 16:17:06.486259 18070 leader_election.cc:290] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 16:17:06.486819 18073 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:06.488607 18073 raft_consensus.cc:697] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 1 LEADER]: Becoming Leader. State: Replica: 387423d8ffe1441b90cc64abbbf247c7, State: Running, Role: LEADER
I20260317 16:17:06.489449 18073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387423d8ffe1441b90cc64abbbf247c7" member_type: VOTER }
I20260317 16:17:06.491063 18070 sys_catalog.cc:565] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 16:17:06.496541 18074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 387423d8ffe1441b90cc64abbbf247c7. Latest consensus state: current_term: 1 leader_uuid: "387423d8ffe1441b90cc64abbbf247c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387423d8ffe1441b90cc64abbbf247c7" member_type: VOTER } }
I20260317 16:17:06.497236 18074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [sys.catalog]: This master's current role is: LEADER
I20260317 16:17:06.497491 18075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "387423d8ffe1441b90cc64abbbf247c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387423d8ffe1441b90cc64abbbf247c7" member_type: VOTER } }
I20260317 16:17:06.498847 18075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [sys.catalog]: This master's current role is: LEADER
I20260317 16:17:06.500156 18079 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 16:17:06.506157 18079 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 16:17:06.517270 16879 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 16:17:06.517395 18079 catalog_manager.cc:1357] Generated new cluster ID: 35a37eb76c3c46c2a8d0ed8754746460
I20260317 16:17:06.517906 18079 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 16:17:06.547175 18079 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 16:17:06.548496 18079 catalog_manager.cc:1514] Loading token signing keys...
I20260317 16:17:06.563253 18079 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7: Generated new TSK 0
I20260317 16:17:06.563920 18079 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 16:17:06.584270 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:17:06.593534 18091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.593672 18092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.595096 18094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:17:06.596167 16879 server_base.cc:1060] running on GCE node
I20260317 16:17:06.597289 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:17:06.597487 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:17:06.597802 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764226597783 us; error 0 us; skew 500 ppm
I20260317 16:17:06.598488 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:17:06.601351 16879 webserver.cc:492] Webserver started at http://127.16.123.193:39671/ using document root <none> and password file <none>
I20260317 16:17:06.601912 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:17:06.602115 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:17:06.602420 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:17:06.604269 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/instance:
uuid: "94edf207909c4f8990bfe29ed1a87a6d"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.609320 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260317 16:17:06.613232 18099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.614298 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260317 16:17:06.614710 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root
uuid: "94edf207909c4f8990bfe29ed1a87a6d"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.615120 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:17:06.633055 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:17:06.634274 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:17:06.635963 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:17:06.638248 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:17:06.638427 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.638705 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:17:06.638933 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260317 16:17:06.684070 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.193:45673
I20260317 16:17:06.684140 18161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.193:45673 every 8 connection(s)
I20260317 16:17:06.688925 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:17:06.697217 18166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.697939 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.700702 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:17:06.701877 16879 server_base.cc:1060] running on GCE node
I20260317 16:17:06.703722 18162 heartbeater.cc:344] Connected to a master server at 127.16.123.254:39965
I20260317 16:17:06.703918 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:17:06.704171 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:17:06.704173 18162 heartbeater.cc:461] Registering TS with master...
I20260317 16:17:06.704500 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764226704478 us; error 0 us; skew 500 ppm
I20260317 16:17:06.705261 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:17:06.705261 18162 heartbeater.cc:507] Master 127.16.123.254:39965 requested a full tablet report, sending...
I20260317 16:17:06.708148 18035 ts_manager.cc:194] Registered new tserver with Master: 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:06.708386 16879 webserver.cc:492] Webserver started at http://127.16.123.194:46815/ using document root <none> and password file <none>
I20260317 16:17:06.709184 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:17:06.709499 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:17:06.709800 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:17:06.710372 18035 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51552
I20260317 16:17:06.711520 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/instance:
uuid: "88b3a81306894b71bf2eacc88c770eaa"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.717115 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260317 16:17:06.720849 18174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.721894 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260317 16:17:06.722215 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root
uuid: "88b3a81306894b71bf2eacc88c770eaa"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.722499 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:17:06.758460 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:17:06.759789 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:17:06.761281 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:17:06.763542 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:17:06.763777 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.764098 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:17:06.764324 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.808171 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.194:35635
I20260317 16:17:06.808295 18236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.194:35635 every 8 connection(s)
I20260317 16:17:06.813256 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:17:06.821501 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.822137 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:06.826252 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:17:06.826409 18237 heartbeater.cc:344] Connected to a master server at 127.16.123.254:39965
I20260317 16:17:06.826916 18237 heartbeater.cc:461] Registering TS with master...
I20260317 16:17:06.826953 16879 server_base.cc:1060] running on GCE node
I20260317 16:17:06.828030 18237 heartbeater.cc:507] Master 127.16.123.254:39965 requested a full tablet report, sending...
I20260317 16:17:06.828107 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:17:06.828426 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:17:06.828635 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764226828617 us; error 0 us; skew 500 ppm
I20260317 16:17:06.829396 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:17:06.830291 18035 ts_manager.cc:194] Registered new tserver with Master: 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:06.831897 18035 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51564
I20260317 16:17:06.832190 16879 webserver.cc:492] Webserver started at http://127.16.123.195:44593/ using document root <none> and password file <none>
I20260317 16:17:06.832762 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:17:06.832989 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:17:06.833285 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:17:06.834506 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/instance:
uuid: "021d78130690452ba094f4a392fb4378"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.838922 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260317 16:17:06.842250 18248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.843278 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260317 16:17:06.843678 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root
uuid: "021d78130690452ba094f4a392fb4378"
format_stamp: "Formatted at 2026-03-17 16:17:06 on dist-test-slave-kqc6"
I20260317 16:17:06.843981 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:17:06.856678 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:17:06.857894 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:17:06.859350 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:17:06.861600 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:17:06.861846 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.862100 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:17:06.862269 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:06.909953 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.195:44851
I20260317 16:17:06.910035 18310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.195:44851 every 8 connection(s)
I20260317 16:17:06.924741 18311 heartbeater.cc:344] Connected to a master server at 127.16.123.254:39965
I20260317 16:17:06.925166 18311 heartbeater.cc:461] Registering TS with master...
I20260317 16:17:06.925908 18311 heartbeater.cc:507] Master 127.16.123.254:39965 requested a full tablet report, sending...
I20260317 16:17:06.927997 18035 ts_manager.cc:194] Registered new tserver with Master: 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:06.928474 16879 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015269305s
I20260317 16:17:06.929605 18035 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51574
I20260317 16:17:06.957160 18035 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51590:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 16:17:06.965167 18035 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 16:17:07.187222 18127 tablet_service.cc:1511] Processing CreateTablet for tablet 2f120096f51b4b3a992048069e2ae182 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260317 16:17:07.188817 18127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f120096f51b4b3a992048069e2ae182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.189182 18125 tablet_service.cc:1511] Processing CreateTablet for tablet 8284701fdaf640daa689a2534914514c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260317 16:17:07.190601 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8284701fdaf640daa689a2534914514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.220157 18271 tablet_service.cc:1511] Processing CreateTablet for tablet 85f478d86a0949f381f0d88a657ba4e5 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260317 16:17:07.222015 18271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f478d86a0949f381f0d88a657ba4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.225405 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 09f47966c253483393878fddf97ac206 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260317 16:17:07.227978 18199 tablet_service.cc:1511] Processing CreateTablet for tablet e6aca2493a994ed888cd663ddcf97582 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260317 16:17:07.230413 18272 tablet_service.cc:1511] Processing CreateTablet for tablet 09f47966c253483393878fddf97ac206 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260317 16:17:07.233171 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f47966c253483393878fddf97ac206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.187431 18126 tablet_service.cc:1511] Processing CreateTablet for tablet c85ea9795a2c43a38f1a9c3f3741db36 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260317 16:17:07.201974 18119 tablet_service.cc:1511] Processing CreateTablet for tablet 9f14079a4ea34cbda92828ca274134f4 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260317 16:17:07.239944 18119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f14079a4ea34cbda92828ca274134f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.241364 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f47966c253483393878fddf97ac206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.197672 18120 tablet_service.cc:1511] Processing CreateTablet for tablet 87a9551ad4c24868805c90f4fa899ff0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260317 16:17:07.243289 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aca2493a994ed888cd663ddcf97582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.243678 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a9551ad4c24868805c90f4fa899ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.192034 18124 tablet_service.cc:1511] Processing CreateTablet for tablet e6aca2493a994ed888cd663ddcf97582 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260317 16:17:07.247390 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aca2493a994ed888cd663ddcf97582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.248239 18270 tablet_service.cc:1511] Processing CreateTablet for tablet 7aafcd49c9d54f548b25a51935f79523 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260317 16:17:07.249684 18270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aafcd49c9d54f548b25a51935f79523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.193802 18123 tablet_service.cc:1511] Processing CreateTablet for tablet 09f47966c253483393878fddf97ac206 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260317 16:17:07.252362 18123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f47966c253483393878fddf97ac206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.195827 18122 tablet_service.cc:1511] Processing CreateTablet for tablet 85f478d86a0949f381f0d88a657ba4e5 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260317 16:17:07.252724 18269 tablet_service.cc:1511] Processing CreateTablet for tablet 87a9551ad4c24868805c90f4fa899ff0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260317 16:17:07.193811 18121 tablet_service.cc:1511] Processing CreateTablet for tablet 7aafcd49c9d54f548b25a51935f79523 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260317 16:17:07.254154 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a9551ad4c24868805c90f4fa899ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.205093 18118 tablet_service.cc:1511] Processing CreateTablet for tablet 290a7a9eceeb4ea8a8ddb67d3e330082 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260317 16:17:07.254156 18122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f478d86a0949f381f0d88a657ba4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.256757 18268 tablet_service.cc:1511] Processing CreateTablet for tablet 9f14079a4ea34cbda92828ca274134f4 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260317 16:17:07.258208 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f14079a4ea34cbda92828ca274134f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.259949 18118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290a7a9eceeb4ea8a8ddb67d3e330082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.261245 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aafcd49c9d54f548b25a51935f79523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.206995 18276 tablet_service.cc:1511] Processing CreateTablet for tablet 2f120096f51b4b3a992048069e2ae182 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260317 16:17:07.208657 18275 tablet_service.cc:1511] Processing CreateTablet for tablet c85ea9795a2c43a38f1a9c3f3741db36 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260317 16:17:07.208770 18193 tablet_service.cc:1511] Processing CreateTablet for tablet 290a7a9eceeb4ea8a8ddb67d3e330082 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260317 16:17:07.210546 18202 tablet_service.cc:1511] Processing CreateTablet for tablet 2f120096f51b4b3a992048069e2ae182 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260317 16:17:07.210541 18194 tablet_service.cc:1511] Processing CreateTablet for tablet 9f14079a4ea34cbda92828ca274134f4 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260317 16:17:07.263054 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f120096f51b4b3a992048069e2ae182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.263659 18193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290a7a9eceeb4ea8a8ddb67d3e330082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.264902 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ea9795a2c43a38f1a9c3f3741db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.265151 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f14079a4ea34cbda92828ca274134f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.266561 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f120096f51b4b3a992048069e2ae182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.267360 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ea9795a2c43a38f1a9c3f3741db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.211756 18274 tablet_service.cc:1511] Processing CreateTablet for tablet 8284701fdaf640daa689a2534914514c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260317 16:17:07.269050 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8284701fdaf640daa689a2534914514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.213658 18201 tablet_service.cc:1511] Processing CreateTablet for tablet c85ea9795a2c43a38f1a9c3f3741db36 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260317 16:17:07.271966 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ea9795a2c43a38f1a9c3f3741db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.214141 18195 tablet_service.cc:1511] Processing CreateTablet for tablet 87a9551ad4c24868805c90f4fa899ff0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260317 16:17:07.273851 18195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a9551ad4c24868805c90f4fa899ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.215564 18273 tablet_service.cc:1511] Processing CreateTablet for tablet e6aca2493a994ed888cd663ddcf97582 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260317 16:17:07.215942 18196 tablet_service.cc:1511] Processing CreateTablet for tablet 7aafcd49c9d54f548b25a51935f79523 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260317 16:17:07.217483 18200 tablet_service.cc:1511] Processing CreateTablet for tablet 8284701fdaf640daa689a2534914514c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260317 16:17:07.275938 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aca2493a994ed888cd663ddcf97582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.217691 18197 tablet_service.cc:1511] Processing CreateTablet for tablet 85f478d86a0949f381f0d88a657ba4e5 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260317 16:17:07.275938 18196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aafcd49c9d54f548b25a51935f79523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.277530 18197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f478d86a0949f381f0d88a657ba4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.278823 18200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8284701fdaf640daa689a2534914514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.325601 18267 tablet_service.cc:1511] Processing CreateTablet for tablet 290a7a9eceeb4ea8a8ddb67d3e330082 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260317 16:17:07.327638 18267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290a7a9eceeb4ea8a8ddb67d3e330082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.384851 18124 tablet_service.cc:1511] Processing CreateTablet for tablet fb31cad8b88240acb9c101027ea444c1 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260317 16:17:07.385165 18127 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf03ff443ed41d3bb50d992c1f34372 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260317 16:17:07.386269 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb31cad8b88240acb9c101027ea444c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.388504 18125 tablet_service.cc:1511] Processing CreateTablet for tablet dbff882fd6aa4aa0a32d72d1674abc76 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260317 16:17:07.389907 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbff882fd6aa4aa0a32d72d1674abc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.394392 18121 tablet_service.cc:1511] Processing CreateTablet for tablet 84ded5b9676242a18139f88dc63af2b4 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260317 16:17:07.395901 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ded5b9676242a18139f88dc63af2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.400002 18126 tablet_service.cc:1511] Processing CreateTablet for tablet 4a773cd94be84692a95e55317c98a95c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260317 16:17:07.401414 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a773cd94be84692a95e55317c98a95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.410462 18326 tablet_bootstrap.cc:492] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:07.447815 18127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf03ff443ed41d3bb50d992c1f34372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.456465 18118 tablet_service.cc:1511] Processing CreateTablet for tablet ab7a4647163b461f87a0a249491e566c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260317 16:17:07.462013 18121 tablet_service.cc:1511] Processing CreateTablet for tablet 24063e0e6d6a4826899168d2524d5abc (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260317 16:17:07.468149 18126 tablet_service.cc:1511] Processing CreateTablet for tablet 9545c351b28449c2a44da8b9a94c7f11 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260317 16:17:07.481355 18202 tablet_service.cc:1511] Processing CreateTablet for tablet fb31cad8b88240acb9c101027ea444c1 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260317 16:17:07.483655 18276 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf03ff443ed41d3bb50d992c1f34372 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260317 16:17:07.485496 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf03ff443ed41d3bb50d992c1f34372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.512086 18200 tablet_service.cc:1511] Processing CreateTablet for tablet dbff882fd6aa4aa0a32d72d1674abc76 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260317 16:17:07.509063 18193 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf03ff443ed41d3bb50d992c1f34372 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260317 16:17:07.518189 18274 tablet_service.cc:1511] Processing CreateTablet for tablet dbff882fd6aa4aa0a32d72d1674abc76 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
W20260317 16:17:07.524776 18021 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 43975.1us
I20260317 16:17:07.532915 18118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7a4647163b461f87a0a249491e566c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.534552 18273 tablet_service.cc:1511] Processing CreateTablet for tablet 84ded5b9676242a18139f88dc63af2b4 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260317 16:17:07.538846 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ded5b9676242a18139f88dc63af2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.542250 18197 tablet_service.cc:1511] Processing CreateTablet for tablet ab7a4647163b461f87a0a249491e566c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260317 16:17:07.539620 18196 tablet_service.cc:1511] Processing CreateTablet for tablet 84ded5b9676242a18139f88dc63af2b4 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260317 16:17:07.548041 18268 tablet_service.cc:1511] Processing CreateTablet for tablet fb31cad8b88240acb9c101027ea444c1 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260317 16:17:07.550103 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb31cad8b88240acb9c101027ea444c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:17:07.558174 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:07.560792 18194 tablet_service.cc:1511] Processing CreateTablet for tablet 701d2829bc6a450a94470d7f852ecdc8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260317 16:17:07.563021 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24063e0e6d6a4826899168d2524d5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.564963 18120 tablet_service.cc:1511] Processing CreateTablet for tablet 701d2829bc6a450a94470d7f852ecdc8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260317 16:17:07.566278 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701d2829bc6a450a94470d7f852ecdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.566072 18125 tablet_service.cc:1511] Processing CreateTablet for tablet 9e145759d6de4334931c0039a97abdd8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
W20260317 16:17:07.567098 18022 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 84486.6us
I20260317 16:17:07.568296 18276 tablet_service.cc:1511] Processing CreateTablet for tablet ab7a4647163b461f87a0a249491e566c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260317 16:17:07.568838 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9545c351b28449c2a44da8b9a94c7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.569077 18275 tablet_service.cc:1511] Processing CreateTablet for tablet 24063e0e6d6a4826899168d2524d5abc (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260317 16:17:07.569563 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7a4647163b461f87a0a249491e566c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.570520 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24063e0e6d6a4826899168d2524d5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.571136 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbff882fd6aa4aa0a32d72d1674abc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.575757 18195 tablet_service.cc:1511] Processing CreateTablet for tablet 4a773cd94be84692a95e55317c98a95c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260317 16:17:07.579138 18326 tablet_bootstrap.cc:654] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:07.523414 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb31cad8b88240acb9c101027ea444c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.584707 18327 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:07.554919 18199 tablet_service.cc:1511] Processing CreateTablet for tablet 24063e0e6d6a4826899168d2524d5abc (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260317 16:17:07.591080 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e145759d6de4334931c0039a97abdd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.596577 18123 tablet_service.cc:1511] Processing CreateTablet for tablet 07c5d59b5b404eec925d5c4681953edb (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260317 16:17:07.600103 18269 tablet_service.cc:1511] Processing CreateTablet for tablet fe3f3aabd9b444868c9732a8e5fa7440 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260317 16:17:07.599637 18271 tablet_service.cc:1511] Processing CreateTablet for tablet 701d2829bc6a450a94470d7f852ecdc8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260317 16:17:07.602620 18328 tablet_bootstrap.cc:492] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:07.612622 18272 tablet_service.cc:1511] Processing CreateTablet for tablet 4a773cd94be84692a95e55317c98a95c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260317 16:17:07.614324 18201 tablet_service.cc:1511] Processing CreateTablet for tablet 9e145759d6de4334931c0039a97abdd8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260317 16:17:07.617363 18270 tablet_service.cc:1511] Processing CreateTablet for tablet 9545c351b28449c2a44da8b9a94c7f11 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260317 16:17:07.619014 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3f3aabd9b444868c9732a8e5fa7440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.622510 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a773cd94be84692a95e55317c98a95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.627544 18271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701d2829bc6a450a94470d7f852ecdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.627928 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24063e0e6d6a4826899168d2524d5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.628746 18200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbff882fd6aa4aa0a32d72d1674abc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.554884 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 9545c351b28449c2a44da8b9a94c7f11 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260317 16:17:07.630741 18267 tablet_service.cc:1511] Processing CreateTablet for tablet 6b643cd863d14f36a3b9341312ef4770 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260317 16:17:07.636559 18326 tablet_bootstrap.cc:492] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:07.636493 18122 tablet_service.cc:1511] Processing CreateTablet for tablet 5ded1f55ede142cbbde7c675a0f72d15 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260317 16:17:07.638136 18270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9545c351b28449c2a44da8b9a94c7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.631816 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9545c351b28449c2a44da8b9a94c7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.647959 18326 ts_tablet_manager.cc:1403] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.238s user 0.016s sys 0.004s
I20260317 16:17:07.660259 18326 raft_consensus.cc:359] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.661049 18326 raft_consensus.cc:385] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:07.661339 18326 raft_consensus.cc:740] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:07.662065 18326 consensus_queue.cc:260] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.662232 18276 tablet_service.cc:1511] Processing CreateTablet for tablet 07c5d59b5b404eec925d5c4681953edb (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260317 16:17:07.674685 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e145759d6de4334931c0039a97abdd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.675098 18199 tablet_service.cc:1511] Processing CreateTablet for tablet fe3f3aabd9b444868c9732a8e5fa7440 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260317 16:17:07.680841 18267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b643cd863d14f36a3b9341312ef4770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.690672 18197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7a4647163b461f87a0a249491e566c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.689374 18327 tablet_bootstrap.cc:654] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:07.691313 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3f3aabd9b444868c9732a8e5fa7440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.695152 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c5d59b5b404eec925d5c4681953edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.695286 18195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a773cd94be84692a95e55317c98a95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.696209 18328 tablet_bootstrap.cc:654] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:07.696596 18127 tablet_service.cc:1511] Processing CreateTablet for tablet fdf3aa784b554e5386dcc11e2550e2f5 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260317 16:17:07.697817 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701d2829bc6a450a94470d7f852ecdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.694806 18123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c5d59b5b404eec925d5c4681953edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.711092 18193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf03ff443ed41d3bb50d992c1f34372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.718858 18162 heartbeater.cc:499] Master 127.16.123.254:39965 was elected leader, sending a full tablet report...
I20260317 16:17:07.723062 18122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ded1f55ede142cbbde7c675a0f72d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.727226 18196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ded5b9676242a18139f88dc63af2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.741627 18202 tablet_service.cc:1511] Processing CreateTablet for tablet 6b643cd863d14f36a3b9341312ef4770 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260317 16:17:07.754372 18124 tablet_service.cc:1511] Processing CreateTablet for tablet 6b643cd863d14f36a3b9341312ef4770 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260317 16:17:07.755790 18118 tablet_service.cc:1511] Processing CreateTablet for tablet fdb770e745e0447faf1334e5d343ace0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260317 16:17:07.750166 18119 tablet_service.cc:1511] Processing CreateTablet for tablet fe3f3aabd9b444868c9732a8e5fa7440 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260317 16:17:07.770921 18127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf3aa784b554e5386dcc11e2550e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.779347 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b643cd863d14f36a3b9341312ef4770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.783097 18119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3f3aabd9b444868c9732a8e5fa7440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.783986 18126 tablet_service.cc:1511] Processing CreateTablet for tablet 0b2a4d319ba9480b9c950e12b7329101 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260317 16:17:07.785369 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2a4d319ba9480b9c950e12b7329101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.791103 18118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb770e745e0447faf1334e5d343ace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.794698 18120 tablet_service.cc:1511] Processing CreateTablet for tablet 96d59cc0e53747428c5ff7f5178f11ca (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260317 16:17:07.796270 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d59cc0e53747428c5ff7f5178f11ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.756907 18273 tablet_service.cc:1511] Processing CreateTablet for tablet 9e145759d6de4334931c0039a97abdd8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260317 16:17:07.800851 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e145759d6de4334931c0039a97abdd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.816234 18199 tablet_service.cc:1511] Processing CreateTablet for tablet 07c5d59b5b404eec925d5c4681953edb (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260317 16:17:07.819990 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b643cd863d14f36a3b9341312ef4770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.834389 18237 heartbeater.cc:499] Master 127.16.123.254:39965 was elected leader, sending a full tablet report...
I20260317 16:17:07.851794 18328 tablet_bootstrap.cc:492] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:07.860407 18328 ts_tablet_manager.cc:1403] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.258s user 0.013s sys 0.009s
I20260317 16:17:07.847462 18121 tablet_service.cc:1511] Processing CreateTablet for tablet 17fe426def0d4f378a44ce5ec328f2e0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260317 16:17:07.865090 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fe426def0d4f378a44ce5ec328f2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.846715 18272 tablet_service.cc:1511] Processing CreateTablet for tablet fdb770e745e0447faf1334e5d343ace0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260317 16:17:07.889760 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb770e745e0447faf1334e5d343ace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.865216 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c5d59b5b404eec925d5c4681953edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.891261 18122 tablet_service.cc:1511] Processing CreateTablet for tablet 44eefa978130466ead1b5828a0e0c4f3 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260317 16:17:07.891791 18200 tablet_service.cc:1511] Processing CreateTablet for tablet 96d59cc0e53747428c5ff7f5178f11ca (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260317 16:17:07.867879 18328 raft_consensus.cc:359] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.893392 18328 raft_consensus.cc:385] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:07.893723 18328 raft_consensus.cc:740] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:07.893844 18200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d59cc0e53747428c5ff7f5178f11ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.894578 18328 consensus_queue.cc:260] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.855674 18326 ts_tablet_manager.cc:1434] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.205s user 0.035s sys 0.016s
I20260317 16:17:07.896611 18326 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:07.876106 18327 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:07.897238 18327 ts_tablet_manager.cc:1403] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.313s user 0.010s sys 0.014s
I20260317 16:17:07.899925 18327 raft_consensus.cc:359] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.900624 18327 raft_consensus.cc:385] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:07.900915 18327 raft_consensus.cc:740] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:07.901309 18275 tablet_service.cc:1511] Processing CreateTablet for tablet 898420a8ccf84bbdba91710ca9c08ef1 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260317 16:17:07.852345 18268 tablet_service.cc:1511] Processing CreateTablet for tablet 5ded1f55ede142cbbde7c675a0f72d15 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260317 16:17:07.871528 18276 tablet_service.cc:1511] Processing CreateTablet for tablet fdf3aa784b554e5386dcc11e2550e2f5 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260317 16:17:07.886731 18193 tablet_service.cc:1511] Processing CreateTablet for tablet 5ded1f55ede142cbbde7c675a0f72d15 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260317 16:17:07.923678 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ded1f55ede142cbbde7c675a0f72d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.923599 18202 tablet_service.cc:1511] Processing CreateTablet for tablet 07a6f1b891ac4c3492ae2548ff2fc357 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260317 16:17:07.924588 18193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ded1f55ede142cbbde7c675a0f72d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.890662 18126 tablet_service.cc:1511] Processing CreateTablet for tablet 9844da16b03545c5a9e5534b29b4343c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260317 16:17:07.925209 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a6f1b891ac4c3492ae2548ff2fc357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.865706 18194 tablet_service.cc:1511] Processing CreateTablet for tablet fdb770e745e0447faf1334e5d343ace0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260317 16:17:07.926487 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb770e745e0447faf1334e5d343ace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.927457 18273 tablet_service.cc:1511] Processing CreateTablet for tablet 47801c6cdf6f461181a89246716390d7 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260317 16:17:07.928696 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47801c6cdf6f461181a89246716390d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.931160 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf3aa784b554e5386dcc11e2550e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.934300 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898420a8ccf84bbdba91710ca9c08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.935077 18311 heartbeater.cc:499] Master 127.16.123.254:39965 was elected leader, sending a full tablet report...
I20260317 16:17:07.937287 18201 tablet_service.cc:1511] Processing CreateTablet for tablet 47801c6cdf6f461181a89246716390d7 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260317 16:17:07.892607 18122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44eefa978130466ead1b5828a0e0c4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.961229 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47801c6cdf6f461181a89246716390d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.901752 18327 consensus_queue.cc:260] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.852562 18267 tablet_service.cc:1511] Processing CreateTablet for tablet 0b2a4d319ba9480b9c950e12b7329101 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260317 16:17:07.893872 18124 tablet_service.cc:1511] Processing CreateTablet for tablet 55ecf327ed214b02bef87b8c05d54c26 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260317 16:17:07.904249 18195 tablet_service.cc:1511] Processing CreateTablet for tablet 17fe426def0d4f378a44ce5ec328f2e0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260317 16:17:07.880362 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 0b2a4d319ba9480b9c950e12b7329101 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260317 16:17:07.862265 18196 tablet_service.cc:1511] Processing CreateTablet for tablet fdf3aa784b554e5386dcc11e2550e2f5 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260317 16:17:07.873152 18270 tablet_service.cc:1511] Processing CreateTablet for tablet 17fe426def0d4f378a44ce5ec328f2e0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260317 16:17:07.909261 18271 tablet_service.cc:1511] Processing CreateTablet for tablet 07a6f1b891ac4c3492ae2548ff2fc357 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260317 16:17:07.876286 18125 tablet_service.cc:1511] Processing CreateTablet for tablet 07a6f1b891ac4c3492ae2548ff2fc357 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260317 16:17:07.963606 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9844da16b03545c5a9e5534b29b4343c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.969429 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a6f1b891ac4c3492ae2548ff2fc357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.963624 18267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2a4d319ba9480b9c950e12b7329101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.914440 18119 tablet_service.cc:1511] Processing CreateTablet for tablet 898420a8ccf84bbdba91710ca9c08ef1 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260317 16:17:07.914712 18197 tablet_service.cc:1511] Processing CreateTablet for tablet 898420a8ccf84bbdba91710ca9c08ef1 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260317 16:17:07.966163 18330 raft_consensus.cc:493] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:07.916369 18123 tablet_service.cc:1511] Processing CreateTablet for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260317 16:17:07.975674 18119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898420a8ccf84bbdba91710ca9c08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.975484 18330 raft_consensus.cc:515] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:07.976339 18123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.978183 18330 leader_election.cc:290] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:07.917320 18328 ts_tablet_manager.cc:1434] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.057s user 0.007s sys 0.001s
I20260317 16:17:07.888406 18127 tablet_service.cc:1511] Processing CreateTablet for tablet 47801c6cdf6f461181a89246716390d7 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260317 16:17:07.869109 18269 tablet_service.cc:1511] Processing CreateTablet for tablet 96d59cc0e53747428c5ff7f5178f11ca (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260317 16:17:07.940790 18121 tablet_service.cc:1511] Processing CreateTablet for tablet 9246439ff23646f8abdf7bba7f3a9172 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260317 16:17:07.943899 18274 tablet_service.cc:1511] Processing CreateTablet for tablet 9844da16b03545c5a9e5534b29b4343c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260317 16:17:07.948967 18273 tablet_service.cc:1511] Processing CreateTablet for tablet 44eefa978130466ead1b5828a0e0c4f3 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260317 16:17:07.957185 18202 tablet_service.cc:1511] Processing CreateTablet for tablet 9844da16b03545c5a9e5534b29b4343c (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260317 16:17:07.959868 18199 tablet_service.cc:1511] Processing CreateTablet for tablet 44eefa978130466ead1b5828a0e0c4f3 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260317 16:17:07.964879 18195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fe426def0d4f378a44ce5ec328f2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.980871 18328 tablet_bootstrap.cc:492] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:07.962802 18120 tablet_service.cc:1511] Processing CreateTablet for tablet ac85b776838146d9904478a4ef0ae824 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260317 16:17:07.999435 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ecf327ed214b02bef87b8c05d54c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:07.984012 18270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fe426def0d4f378a44ce5ec328f2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.012415 18272 tablet_service.cc:1511] Processing CreateTablet for tablet 55ecf327ed214b02bef87b8c05d54c26 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260317 16:17:08.026026 18333 raft_consensus.cc:493] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:08.026612 18333 raft_consensus.cc:515] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.029547 18333 leader_election.cc:290] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:08.031320 18196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdf3aa784b554e5386dcc11e2550e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.025614 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9844da16b03545c5a9e5534b29b4343c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.040833 18127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47801c6cdf6f461181a89246716390d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.061051 18326 tablet_bootstrap.cc:654] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.075250 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2a4d319ba9480b9c950e12b7329101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.087517 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44eefa978130466ead1b5828a0e0c4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.090827 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85b776838146d9904478a4ef0ae824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.097873 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260317 16:17:08.099262 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ecf327ed214b02bef87b8c05d54c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.124575 18118 tablet_service.cc:1511] Processing CreateTablet for tablet d88548519a0f473680897b8a7fc1e9b6 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260317 16:17:08.135581 18271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a6f1b891ac4c3492ae2548ff2fc357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.137162 18268 tablet_service.cc:1511] Processing CreateTablet for tablet 9246439ff23646f8abdf7bba7f3a9172 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260317 16:17:08.138463 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9246439ff23646f8abdf7bba7f3a9172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.111385 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9246439ff23646f8abdf7bba7f3a9172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.113301 18197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898420a8ccf84bbdba91710ca9c08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.090546 18194 tablet_service.cc:1511] Processing CreateTablet for tablet 55ecf327ed214b02bef87b8c05d54c26 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260317 16:17:08.146104 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d59cc0e53747428c5ff7f5178f11ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.147838 18275 tablet_service.cc:1511] Processing CreateTablet for tablet ac85b776838146d9904478a4ef0ae824 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260317 16:17:08.148048 18119 tablet_service.cc:1511] Processing CreateTablet for tablet c94382dceadb48228dd3467bf673a11a (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260317 16:17:08.159662 18327 ts_tablet_manager.cc:1434] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.262s user 0.016s sys 0.010s
I20260317 16:17:08.162099 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9844da16b03545c5a9e5534b29b4343c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.167173 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44eefa978130466ead1b5828a0e0c4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.166958 18123 tablet_service.cc:1511] Processing CreateTablet for tablet 4ee9125e28d14f3c8f458dda86c4ee15 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260317 16:17:08.171376 18276 tablet_service.cc:1511] Processing CreateTablet for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260317 16:17:08.184103 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85b776838146d9904478a4ef0ae824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.181144 18118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88548519a0f473680897b8a7fc1e9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.191447 18327 tablet_bootstrap.cc:492] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:08.205590 18327 tablet_bootstrap.cc:654] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.207101 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.210322 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ecf327ed214b02bef87b8c05d54c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.215982 18125 tablet_service.cc:1511] Processing CreateTablet for tablet 94f2b31d108e461d932c2477e0f78fd0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260317 16:17:08.217443 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2b31d108e461d932c2477e0f78fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.217631 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.220633 18122 tablet_service.cc:1511] Processing CreateTablet for tablet 70f2fc6f0fd640fcb531a01d7881a614 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260317 16:17:08.221968 18123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9125e28d14f3c8f458dda86c4ee15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.224177 18122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f2fc6f0fd640fcb531a01d7881a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.227556 18119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94382dceadb48228dd3467bf673a11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.240260 18328 tablet_bootstrap.cc:654] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.259212 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:08.261183 18102 leader_election.cc:343] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.259126 18200 tablet_service.cc:1511] Processing CreateTablet for tablet 9246439ff23646f8abdf7bba7f3a9172 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260317 16:17:08.262521 18200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9246439ff23646f8abdf7bba7f3a9172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.266907 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:08.268517 18175 leader_election.cc:343] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.272972 18120 tablet_service.cc:1511] Processing CreateTablet for tablet e390c786df0b48878244a08e889950a3 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260317 16:17:08.276404 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:08.278949 18103 leader_election.cc:343] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.279453 18103 leader_election.cc:304] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:08.279511 18201 tablet_service.cc:1511] Processing CreateTablet for tablet d88548519a0f473680897b8a7fc1e9b6 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260317 16:17:08.280848 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88548519a0f473680897b8a7fc1e9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.281145 18330 raft_consensus.cc:2749] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:08.283176 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:08.284264 18202 tablet_service.cc:1511] Processing CreateTablet for tablet ac85b776838146d9904478a4ef0ae824 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260317 16:17:08.285651 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85b776838146d9904478a4ef0ae824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.287746 18193 tablet_service.cc:1511] Processing CreateTablet for tablet c94382dceadb48228dd3467bf673a11a (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260317 16:17:08.289057 18193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94382dceadb48228dd3467bf673a11a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:17:08.291093 18178 leader_election.cc:343] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.291625 18178 leader_election.cc:304] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:08.296846 18274 tablet_service.cc:1511] Processing CreateTablet for tablet c94382dceadb48228dd3467bf673a11a (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260317 16:17:08.298236 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94382dceadb48228dd3467bf673a11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.299355 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e390c786df0b48878244a08e889950a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.321925 18121 tablet_service.cc:1511] Processing CreateTablet for tablet 77f02a05d0ee4e73bafe3055cfcf02f2 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260317 16:17:08.307075 18326 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:08.357683 18326 ts_tablet_manager.cc:1403] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.461s user 0.027s sys 0.007s
I20260317 16:17:08.357252 18267 tablet_service.cc:1511] Processing CreateTablet for tablet e390c786df0b48878244a08e889950a3 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260317 16:17:08.358960 18267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e390c786df0b48878244a08e889950a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.360596 18326 raft_consensus.cc:359] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.346216 18269 tablet_service.cc:1511] Processing CreateTablet for tablet 70f2fc6f0fd640fcb531a01d7881a614 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260317 16:17:08.353961 18268 tablet_service.cc:1511] Processing CreateTablet for tablet 94f2b31d108e461d932c2477e0f78fd0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260317 16:17:08.362835 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f2fc6f0fd640fcb531a01d7881a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.310953 18118 tablet_service.cc:1511] Processing CreateTablet for tablet b451ac456d424629ae5ce6040cb15bd2 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260317 16:17:08.363930 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2b31d108e461d932c2477e0f78fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.304905 18333 raft_consensus.cc:2749] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:08.361322 18326 raft_consensus.cc:385] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.367502 18326 raft_consensus.cc:740] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:08.368368 18326 consensus_queue.cc:260] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.368769 18272 tablet_service.cc:1511] Processing CreateTablet for tablet 1897e8c1602b4aeda93d9b8207b3ac45 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260317 16:17:08.372036 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1897e8c1602b4aeda93d9b8207b3ac45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.368772 18274 tablet_service.cc:1511] Processing CreateTablet for tablet b451ac456d424629ae5ce6040cb15bd2 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260317 16:17:08.374485 18334 raft_consensus.cc:493] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:08.348659 18199 tablet_service.cc:1511] Processing CreateTablet for tablet 70f2fc6f0fd640fcb531a01d7881a614 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260317 16:17:08.375821 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b451ac456d424629ae5ce6040cb15bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.375530 18334 raft_consensus.cc:515] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.376870 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f2fc6f0fd640fcb531a01d7881a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.380157 18334 leader_election.cc:290] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:08.315800 18271 tablet_service.cc:1511] Processing CreateTablet for tablet 4ee9125e28d14f3c8f458dda86c4ee15 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260317 16:17:08.328559 18327 tablet_bootstrap.cc:492] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:08.314242 18195 tablet_service.cc:1511] Processing CreateTablet for tablet 4ee9125e28d14f3c8f458dda86c4ee15 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260317 16:17:08.298753 18275 tablet_service.cc:1511] Processing CreateTablet for tablet d88548519a0f473680897b8a7fc1e9b6 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260317 16:17:08.391098 18271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9125e28d14f3c8f458dda86c4ee15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.316906 18127 tablet_service.cc:1511] Processing CreateTablet for tablet f162d501437442c1ad448749ff6b9d77 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260317 16:17:08.311861 18126 tablet_service.cc:1511] Processing CreateTablet for tablet f77a63cdb5604ab493b7f5f9dc921763 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260317 16:17:08.405048 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 94f2b31d108e461d932c2477e0f78fd0 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260317 16:17:08.406390 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2b31d108e461d932c2477e0f78fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.410029 18195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9125e28d14f3c8f458dda86c4ee15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.421875 18326 ts_tablet_manager.cc:1434] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.064s user 0.002s sys 0.005s
I20260317 16:17:08.423748 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88548519a0f473680897b8a7fc1e9b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.425115 18328 tablet_bootstrap.cc:492] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:08.425621 18328 ts_tablet_manager.cc:1403] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.446s user 0.028s sys 0.020s
I20260317 16:17:08.415550 18327 ts_tablet_manager.cc:1403] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.224s user 0.007s sys 0.011s
I20260317 16:17:08.428490 18328 raft_consensus.cc:359] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.436554 18328 raft_consensus.cc:385] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.436878 18328 raft_consensus.cc:740] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:08.437722 18328 consensus_queue.cc:260] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.440213 18328 ts_tablet_manager.cc:1434] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20260317 16:17:08.441212 18328 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:08.433923 18327 raft_consensus.cc:359] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.443652 18327 raft_consensus.cc:385] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.443966 18327 raft_consensus.cc:740] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
W20260317 16:17:08.444531 18163 tablet.cc:2379] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 16:17:08.444737 18327 consensus_queue.cc:260] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.434264 18326 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:08.426723 18124 tablet_service.cc:1511] Processing CreateTablet for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260317 16:17:08.435581 18268 tablet_service.cc:1511] Processing CreateTablet for tablet e2a036705e4a4da5a5392d119edb244e (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260317 16:17:08.448791 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a036705e4a4da5a5392d119edb244e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.449574 18270 tablet_service.cc:1511] Processing CreateTablet for tablet f77a63cdb5604ab493b7f5f9dc921763 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260317 16:17:08.416080 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f02a05d0ee4e73bafe3055cfcf02f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.453220 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77a63cdb5604ab493b7f5f9dc921763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.453862 18118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b451ac456d424629ae5ce6040cb15bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.454484 18127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f162d501437442c1ad448749ff6b9d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.455953 18122 tablet_service.cc:1511] Processing CreateTablet for tablet e2a036705e4a4da5a5392d119edb244e (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260317 16:17:08.461905 18197 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57484 (request call id 29) took 1021 ms. Trace:
I20260317 16:17:08.464773 18119 tablet_service.cc:1511] Processing CreateTablet for tablet 1897e8c1602b4aeda93d9b8207b3ac45 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260317 16:17:08.465955 18267 tablet_service.cc:1511] Processing CreateTablet for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260317 16:17:08.467089 18327 ts_tablet_manager.cc:1434] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.036s user 0.005s sys 0.003s
I20260317 16:17:08.469970 18125 tablet_service.cc:1511] Processing CreateTablet for tablet 8d3017628c5244c3a8bb091d0a4768fd (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260317 16:17:08.463215 18197 rpcz_store.cc:276] 0317 16:17:07.440880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:17:07.913807 (+472927us) service_pool.cc:225] Handling call
0317 16:17:07.975291 (+ 61484us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:17:07.975384 (+ 93us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:17:08.328718 (+353334us) tablet_metadata.cc:641] Metadata flushed
0317 16:17:08.461123 (+132405us) tablet_replica.cc:206] Creating consensus instance
0317 16:17:08.461812 (+ 689us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:17:08.473299 18194 tablet_service.cc:1511] Processing CreateTablet for tablet e390c786df0b48878244a08e889950a3 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260317 16:17:08.488826 18270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77a63cdb5604ab493b7f5f9dc921763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.499909 18327 tablet_bootstrap.cc:492] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:08.500507 18276 tablet_service.cc:1511] Processing CreateTablet for tablet f162d501437442c1ad448749ff6b9d77 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260317 16:17:08.501991 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f162d501437442c1ad448749ff6b9d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.503504 18197 tablet_service.cc:1511] Processing CreateTablet for tablet f77a63cdb5604ab493b7f5f9dc921763 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260317 16:17:08.504855 18197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77a63cdb5604ab493b7f5f9dc921763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.505357 18269 tablet_service.cc:1511] Processing CreateTablet for tablet 77f02a05d0ee4e73bafe3055cfcf02f2 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260317 16:17:08.513077 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e390c786df0b48878244a08e889950a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.519719 18196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57484 (request call id 24) took 1157 ms. Trace:
I20260317 16:17:08.520228 18196 rpcz_store.cc:276] 0317 16:17:07.362623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:17:07.861282 (+498659us) service_pool.cc:225] Handling call
0317 16:17:07.965696 (+104414us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:17:07.965858 (+ 162us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:17:08.300192 (+334334us) tablet_metadata.cc:641] Metadata flushed
0317 16:17:08.519159 (+218967us) tablet_replica.cc:206] Creating consensus instance
0317 16:17:08.519648 (+ 489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:17:08.506229 18327 tablet_bootstrap.cc:654] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.534796 18267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.535835 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:08.537501 18251 leader_election.cc:343] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.538288 18328 tablet_bootstrap.cc:654] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.509483 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.539189 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f02a05d0ee4e73bafe3055cfcf02f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.549238 18119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1897e8c1602b4aeda93d9b8207b3ac45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.514498 18123 tablet_service.cc:1511] Processing CreateTablet for tablet a8ed2db954354e669de7909d315142aa (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260317 16:17:08.553468 18196 tablet_service.cc:1511] Processing CreateTablet for tablet 77f02a05d0ee4e73bafe3055cfcf02f2 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260317 16:17:08.551084 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3017628c5244c3a8bb091d0a4768fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.567116 18122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a036705e4a4da5a5392d119edb244e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.555532 18200 tablet_service.cc:1511] Processing CreateTablet for tablet b451ac456d424629ae5ce6040cb15bd2 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260317 16:17:08.568621 18123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ed2db954354e669de7909d315142aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:17:08.560693 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:08.560520 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 8d3017628c5244c3a8bb091d0a4768fd (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260317 16:17:08.577214 18202 tablet_service.cc:1511] Processing CreateTablet for tablet e2a036705e4a4da5a5392d119edb244e (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260317 16:17:08.579032 18196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f02a05d0ee4e73bafe3055cfcf02f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.561165 18201 tablet_service.cc:1511] Processing CreateTablet for tablet f162d501437442c1ad448749ff6b9d77 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260317 16:17:08.582835 18274 tablet_service.cc:1511] Processing CreateTablet for tablet 8d3017628c5244c3a8bb091d0a4768fd (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260317 16:17:08.583910 18334 raft_consensus.cc:493] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:08.584376 18274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3017628c5244c3a8bb091d0a4768fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.584422 18334 raft_consensus.cc:515] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.587057 18200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b451ac456d424629ae5ce6040cb15bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.588155 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:08.588884 18212 raft_consensus.cc:2468] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:08.590433 18251 leader_election.cc:304] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:08.586658 18334 leader_election.cc:290] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:08.591112 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f162d501437442c1ad448749ff6b9d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.602860 18334 raft_consensus.cc:2804] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:08.603287 18334 raft_consensus.cc:493] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:08.603623 18334 raft_consensus.cc:3060] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:08.604390 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:08.603466 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:08.605141 18199 tablet_service.cc:1511] Processing CreateTablet for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260317 16:17:08.606428 18249 leader_election.cc:343] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:08.607424 18249 leader_election.cc:343] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.607915 18249 leader_election.cc:304] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:08.611604 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d3017628c5244c3a8bb091d0a4768fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.615089 18333 raft_consensus.cc:493] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:08.615581 18333 raft_consensus.cc:515] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.617810 18333 leader_election.cc:290] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:08.619712 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:08.620179 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:08.621084 18137 raft_consensus.cc:2468] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
W20260317 16:17:08.621480 18178 leader_election.cc:343] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.618980 18334 raft_consensus.cc:515] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.625224 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:08.625844 18212 raft_consensus.cc:3060] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:08.638526 18349 raft_consensus.cc:2749] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:08.639921 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:08.641152 18121 tablet_service.cc:1511] Processing CreateTablet for tablet c5389f9d0bc44e87a3f5897fb4b550ef (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260317 16:17:08.630375 18334 leader_election.cc:290] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:08.644986 18275 tablet_service.cc:1511] Processing CreateTablet for tablet a8ed2db954354e669de7909d315142aa (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260317 16:17:08.694132 18249 leader_election.cc:343] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.695252 18275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ed2db954354e669de7909d315142aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.660071 18127 tablet_service.cc:1511] Processing CreateTablet for tablet 0eb8a46e02034bd199afdbe545379351 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260317 16:17:08.699460 18126 tablet_service.cc:1511] Processing CreateTablet for tablet f1e740c77f5f4cb6916938c76fd04455 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260317 16:17:08.647429 18269 tablet_service.cc:1511] Processing CreateTablet for tablet f1e740c77f5f4cb6916938c76fd04455 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260317 16:17:08.655891 18193 tablet_service.cc:1511] Processing CreateTablet for tablet 1897e8c1602b4aeda93d9b8207b3ac45 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260317 16:17:08.672792 18199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.674310 18175 leader_election.cc:304] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:08.703926 18333 raft_consensus.cc:2804] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:08.704329 18333 raft_consensus.cc:493] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:08.704664 18333 raft_consensus.cc:3060] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:08.692592 18272 tablet_service.cc:1511] Processing CreateTablet for tablet 0eb8a46e02034bd199afdbe545379351 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260317 16:17:08.706014 18272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb8a46e02034bd199afdbe545379351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.674378 18120 tablet_service.cc:1511] Processing CreateTablet for tablet 519bc447dbc1455c844945ba63c96914 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260317 16:17:08.707053 18202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a036705e4a4da5a5392d119edb244e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.678210 18326 tablet_bootstrap.cc:700] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent opening tablet: real 0.112s user 0.005s sys 0.000s
I20260317 16:17:08.707846 18326 tablet_bootstrap.cc:654] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.707991 18194 tablet_service.cc:1511] Processing CreateTablet for tablet 0eb8a46e02034bd199afdbe545379351 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260317 16:17:08.709215 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb8a46e02034bd199afdbe545379351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.678907 18125 tablet_service.cc:1511] Processing CreateTablet for tablet bfaeab6c75f94e128924d831ee8ef55b (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260317 16:17:08.685387 18212 raft_consensus.cc:2468] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:08.689549 18196 tablet_service.cc:1511] Processing CreateTablet for tablet a8ed2db954354e669de7909d315142aa (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260317 16:17:08.693795 18121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5389f9d0bc44e87a3f5897fb4b550ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.700752 18195 tablet_service.cc:1511] Processing CreateTablet for tablet f1e740c77f5f4cb6916938c76fd04455 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260317 16:17:08.723346 18276 tablet_service.cc:1511] Processing CreateTablet for tablet 519bc447dbc1455c844945ba63c96914 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260317 16:17:08.724224 18195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e740c77f5f4cb6916938c76fd04455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.724493 18327 tablet_bootstrap.cc:492] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:08.725294 18126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e740c77f5f4cb6916938c76fd04455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.725630 18327 ts_tablet_manager.cc:1403] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.226s user 0.022s sys 0.008s
I20260317 16:17:08.724355 18251 leader_election.cc:304] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:08.727108 18125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeab6c75f94e128924d831ee8ef55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.729399 18120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519bc447dbc1455c844945ba63c96914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.729566 18269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e740c77f5f4cb6916938c76fd04455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.731415 18276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519bc447dbc1455c844945ba63c96914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.734021 18327 raft_consensus.cc:359] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.749225 18330 raft_consensus.cc:493] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:08.749934 18330 raft_consensus.cc:515] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.739281 18273 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33376 (request call id 33) took 1143 ms. Trace:
I20260317 16:17:08.751191 18273 rpcz_store.cc:276] 0317 16:17:07.595876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:17:07.948031 (+352155us) service_pool.cc:225] Handling call
0317 16:17:08.017391 (+ 69360us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b58002adb98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e32ead1 00007fea3e30bd63 00007fea3e2ba4c0 00007fea37748bf4 00007fea37748b82 00007fea37748ab1
0317 16:17:08.017562 (+ 171us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:17:08.017669 (+ 107us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:17:08.540681 (+523012us) tablet_metadata.cc:641] Metadata flushed
0317 16:17:08.704471 (+163790us) tablet_replica.cc:206] Creating consensus instance
0317 16:17:08.721976 (+ 17505us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6762,"spinlock_wait_cycles":16045952}
I20260317 16:17:08.752458 18328 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:08.752957 18328 ts_tablet_manager.cc:1403] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.312s user 0.018s sys 0.014s
I20260317 16:17:08.753428 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:08.753955 18273 tablet_service.cc:1511] Processing CreateTablet for tablet 702896c0cf9948d68e4c4148743bc5c8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260317 16:17:08.755223 18102 leader_election.cc:343] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.755822 18267 tablet_service.cc:1511] Processing CreateTablet for tablet c5389f9d0bc44e87a3f5897fb4b550ef (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260317 16:17:08.758042 18127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb8a46e02034bd199afdbe545379351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.737784 18327 raft_consensus.cc:385] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.743283 18349 raft_consensus.cc:2804] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:08.763613 18267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5389f9d0bc44e87a3f5897fb4b550ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.760411 18194 tablet_service.cc:1511] Processing CreateTablet for tablet 519bc447dbc1455c844945ba63c96914 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260317 16:17:08.765023 18330 leader_election.cc:290] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:08.766995 18327 raft_consensus.cc:740] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:08.780351 18333 raft_consensus.cc:515] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.769366 18349 raft_consensus.cc:697] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:08.782141 18349 consensus_queue.cc:237] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.783888 18333 leader_election.cc:290] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:08.783939 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:08.780486 18327 consensus_queue.cc:260] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.791145 18137 raft_consensus.cc:3060] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:08.784073 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:08.789652 18124 tablet_service.cc:1511] Processing CreateTablet for tablet 702896c0cf9948d68e4c4148743bc5c8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260317 16:17:08.766005 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:08.803581 18193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1897e8c1602b4aeda93d9b8207b3ac45. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:17:08.817634 18103 leader_election.cc:343] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:08.818104 18103 leader_election.cc:304] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:08.800827 18285 raft_consensus.cc:3060] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:08.819525 18330 raft_consensus.cc:2749] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:08.831758 18137 raft_consensus.cc:2468] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:08.833045 18175 leader_election.cc:304] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:08.774631 18201 tablet_service.cc:1511] Processing CreateTablet for tablet c5389f9d0bc44e87a3f5897fb4b550ef (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260317 16:17:08.833963 18333 raft_consensus.cc:2804] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:08.839061 18201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5389f9d0bc44e87a3f5897fb4b550ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.839440 18268 tablet_service.cc:1511] Processing CreateTablet for tablet bfaeab6c75f94e128924d831ee8ef55b (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260317 16:17:08.840113 18327 ts_tablet_manager.cc:1434] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.114s user 0.007s sys 0.000s
I20260317 16:17:08.784107 18328 raft_consensus.cc:359] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.842420 18328 raft_consensus.cc:385] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.848271 18196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ed2db954354e669de7909d315142aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.860630 18194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519bc447dbc1455c844945ba63c96914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.863449 18326 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:08.849334 18273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702896c0cf9948d68e4c4148743bc5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.864116 18326 ts_tablet_manager.cc:1403] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.430s user 0.029s sys 0.034s
I20260317 16:17:08.863938 18035 catalog_manager.cc:5671] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:08.867785 18285 raft_consensus.cc:2468] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:08.877712 18197 tablet_service.cc:1511] Processing CreateTablet for tablet bfaeab6c75f94e128924d831ee8ef55b (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260317 16:17:08.878623 18326 raft_consensus.cc:359] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.853734 18124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702896c0cf9948d68e4c4148743bc5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.907044 18326 raft_consensus.cc:385] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.883208 18327 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:08.907510 18326 raft_consensus.cc:740] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:08.884680 18198 tablet_service.cc:1511] Processing CreateTablet for tablet 702896c0cf9948d68e4c4148743bc5c8 (DEFAULT_TABLE table=test-workload [id=d8d01909e6e14a16a1638223e144843f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260317 16:17:08.887293 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeab6c75f94e128924d831ee8ef55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.899744 18333 raft_consensus.cc:697] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:08.908496 18326 consensus_queue.cc:260] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.909103 18333 consensus_queue.cc:237] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.854566 18328 raft_consensus.cc:740] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:08.911501 18328 consensus_queue.cc:260] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.915405 18198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702896c0cf9948d68e4c4148743bc5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.921795 18326 ts_tablet_manager.cc:1434] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.057s user 0.007s sys 0.000s
I20260317 16:17:08.923056 18197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeab6c75f94e128924d831ee8ef55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:08.926183 18327 tablet_bootstrap.cc:654] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.929525 18326 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:08.932343 18328 ts_tablet_manager.cc:1434] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.179s user 0.008s sys 0.008s
I20260317 16:17:08.933591 18328 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:08.948501 18328 tablet_bootstrap.cc:654] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.961257 18033 catalog_manager.cc:5671] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:08.965421 18328 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:08.966023 18328 ts_tablet_manager.cc:1403] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.011s
I20260317 16:17:08.974398 18328 raft_consensus.cc:359] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.975034 18326 tablet_bootstrap.cc:654] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:08.975209 18328 raft_consensus.cc:385] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:08.975594 18328 raft_consensus.cc:740] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:08.976729 18328 consensus_queue.cc:260] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:08.979405 18328 ts_tablet_manager.cc:1434] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260317 16:17:08.980543 18328 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:08.992453 18328 tablet_bootstrap.cc:654] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.001492 18327 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.002310 18327 ts_tablet_manager.cc:1403] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.007s
I20260317 16:17:09.004468 18328 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.005009 18328 ts_tablet_manager.cc:1403] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.004s
I20260317 16:17:09.005174 18327 raft_consensus.cc:359] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.005988 18327 raft_consensus.cc:385] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.006335 18327 raft_consensus.cc:740] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.007856 18327 consensus_queue.cc:260] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.008030 18328 raft_consensus.cc:359] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.008803 18328 raft_consensus.cc:385] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.009230 18326 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.009590 18328 raft_consensus.cc:740] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.009886 18326 ts_tablet_manager.cc:1403] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.005s
I20260317 16:17:09.010699 18328 consensus_queue.cc:260] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.011493 18327 ts_tablet_manager.cc:1434] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260317 16:17:09.014199 18328 ts_tablet_manager.cc:1434] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.009s user 0.006s sys 0.005s
I20260317 16:17:09.014601 18327 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.018186 18328 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.025206 18327 tablet_bootstrap.cc:654] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.026801 18328 tablet_bootstrap.cc:654] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.016088 18326 raft_consensus.cc:359] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.027809 18326 raft_consensus.cc:385] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.028213 18326 raft_consensus.cc:740] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.029062 18326 consensus_queue.cc:260] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.035274 18326 ts_tablet_manager.cc:1434] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260317 16:17:09.036459 18326 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.036512 18328 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.037388 18328 ts_tablet_manager.cc:1403] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.002s
I20260317 16:17:09.040488 18327 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.041534 18327 ts_tablet_manager.cc:1403] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.007s
I20260317 16:17:09.041673 18328 raft_consensus.cc:359] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.043083 18328 raft_consensus.cc:385] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.043529 18328 raft_consensus.cc:740] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.044770 18328 consensus_queue.cc:260] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.045225 18327 raft_consensus.cc:359] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.046016 18327 raft_consensus.cc:385] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.046347 18327 raft_consensus.cc:740] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.047082 18327 consensus_queue.cc:260] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.049464 18327 ts_tablet_manager.cc:1434] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260317 16:17:09.050928 18328 ts_tablet_manager.cc:1434] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.013s user 0.005s sys 0.003s
I20260317 16:17:09.051157 18327 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.051928 18328 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.052629 18326 tablet_bootstrap.cc:654] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.058843 18328 tablet_bootstrap.cc:654] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.063964 18333 raft_consensus.cc:493] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.064484 18333 raft_consensus.cc:515] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.066479 18333 leader_election.cc:290] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.066721 18327 tablet_bootstrap.cc:654] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.068218 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8284701fdaf640daa689a2534914514c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.068728 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8284701fdaf640daa689a2534914514c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.068889 18285 raft_consensus.cc:2468] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.069341 18137 raft_consensus.cc:2468] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.070660 18178 leader_election.cc:304] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:09.072011 18333 raft_consensus.cc:2804] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.072466 18333 raft_consensus.cc:493] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.072875 18333 raft_consensus.cc:3060] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.074306 18326 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.074854 18326 ts_tablet_manager.cc:1403] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.006s
I20260317 16:17:09.075727 18328 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.075727 18327 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.076311 18328 ts_tablet_manager.cc:1403] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.010s
I20260317 16:17:09.076360 18327 ts_tablet_manager.cc:1403] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.004s
I20260317 16:17:09.077677 18326 raft_consensus.cc:359] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.078527 18326 raft_consensus.cc:385] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.078939 18326 raft_consensus.cc:740] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.079309 18328 raft_consensus.cc:359] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.079588 18327 raft_consensus.cc:359] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.080037 18328 raft_consensus.cc:385] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.079757 18326 consensus_queue.cc:260] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.080432 18327 raft_consensus.cc:385] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.080785 18327 raft_consensus.cc:740] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.081429 18327 consensus_queue.cc:260] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.082171 18326 ts_tablet_manager.cc:1434] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:17:09.083240 18326 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.083632 18327 ts_tablet_manager.cc:1434] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260317 16:17:09.080440 18328 raft_consensus.cc:740] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.084728 18328 consensus_queue.cc:260] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.087450 18328 ts_tablet_manager.cc:1434] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260317 16:17:09.088229 18327 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.089175 18328 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.089389 18326 tablet_bootstrap.cc:654] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.095897 18328 tablet_bootstrap.cc:654] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.095896 18327 tablet_bootstrap.cc:654] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.108793 18333 raft_consensus.cc:515] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.112602 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8284701fdaf640daa689a2534914514c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.113185 18137 raft_consensus.cc:3060] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.114588 18328 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.115094 18328 ts_tablet_manager.cc:1403] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.005s
I20260317 16:17:09.117515 18327 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.118392 18327 ts_tablet_manager.cc:1403] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.011s
I20260317 16:17:09.118736 18328 raft_consensus.cc:359] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.119728 18328 raft_consensus.cc:385] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.123459 18328 raft_consensus.cc:740] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.122867 18327 raft_consensus.cc:359] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.124138 18327 raft_consensus.cc:385] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.124259 18328 consensus_queue.cc:260] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.124841 18327 raft_consensus.cc:740] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.126674 18328 ts_tablet_manager.cc:1434] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260317 16:17:09.126088 18327 consensus_queue.cc:260] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.129664 18327 ts_tablet_manager.cc:1434] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260317 16:17:09.130666 18328 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.122900 18137 raft_consensus.cc:2468] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.138543 18327 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.139026 18175 leader_election.cc:304] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.140064 18333 leader_election.cc:290] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.140713 18333 raft_consensus.cc:2804] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.141345 18333 raft_consensus.cc:697] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:09.141963 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8284701fdaf640daa689a2534914514c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.142520 18285 raft_consensus.cc:3060] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.142979 18333 consensus_queue.cc:237] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.147890 18327 tablet_bootstrap.cc:654] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.151755 18285 raft_consensus.cc:2468] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.152941 18328 tablet_bootstrap.cc:654] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.153594 18326 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.154366 18326 ts_tablet_manager.cc:1403] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.071s user 0.032s sys 0.036s
I20260317 16:17:09.159699 18033 catalog_manager.cc:5671] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:09.158380 18326 raft_consensus.cc:359] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.161675 18326 raft_consensus.cc:385] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.162114 18326 raft_consensus.cc:740] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.163172 18326 consensus_queue.cc:260] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.169621 18326 ts_tablet_manager.cc:1434] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260317 16:17:09.170601 18326 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.181433 18326 tablet_bootstrap.cc:654] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.181633 18327 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.182170 18327 ts_tablet_manager.cc:1403] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.051s user 0.029s sys 0.010s
I20260317 16:17:09.183274 18328 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.183761 18328 ts_tablet_manager.cc:1403] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.056s user 0.026s sys 0.005s
I20260317 16:17:09.184751 18327 raft_consensus.cc:359] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.185434 18327 raft_consensus.cc:385] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.185704 18327 raft_consensus.cc:740] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.186448 18327 consensus_queue.cc:260] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.187381 18328 raft_consensus.cc:359] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.188071 18328 raft_consensus.cc:385] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.188320 18328 raft_consensus.cc:740] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.189491 18327 ts_tablet_manager.cc:1434] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260317 16:17:09.189162 18328 consensus_queue.cc:260] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.189920 18326 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.190424 18326 ts_tablet_manager.cc:1403] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.002s
I20260317 16:17:09.190549 18327 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.193568 18326 raft_consensus.cc:359] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.194200 18333 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.194402 18326 raft_consensus.cc:385] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.194829 18326 raft_consensus.cc:740] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.194749 18333 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.195879 18326 consensus_queue.cc:260] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.196810 18333 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.197356 18327 tablet_bootstrap.cc:654] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.198452 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.194999 18349 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.199712 18349 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.200217 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:09.201542 18175 leader_election.cc:343] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.201694 18349 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.202005 18326 ts_tablet_manager.cc:1434] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260317 16:17:09.202600 18285 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.203429 18326 tablet_bootstrap.cc:492] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.204695 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.204805 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.205667 18212 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:09.206525 18178 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
W20260317 16:17:09.207149 18249 leader_election.cc:343] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.207643 18333 raft_consensus.cc:2804] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.208027 18333 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.208428 18333 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.210163 18353 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.208122 18251 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.211185 18326 tablet_bootstrap.cc:654] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.210789 18353 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.212010 18349 raft_consensus.cc:2804] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.212656 18349 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.212914 18353 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.210213 18330 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.213095 18349 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.219110 18328 ts_tablet_manager.cc:1434] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.035s user 0.005s sys 0.002s
I20260317 16:17:09.219615 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.213660 18330 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.220659 18328 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.221835 18330 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.223572 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.224257 18212 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:09.227856 18328 tablet_bootstrap.cc:654] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.226238 18137 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.228387 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.229333 18102 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.231065 18330 raft_consensus.cc:2804] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.231124 18175 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
W20260317 16:17:09.231946 18103 leader_election.cc:343] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.232227 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.232391 18353 raft_consensus.cc:2804] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.232889 18353 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.233086 18333 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.233676 18330 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:17:09.233785 18178 leader_election.cc:343] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.234412 18330 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.233376 18353 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:09.239373 18249 consensus_peers.cc:597] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:09.234469 18349 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.246188 18334 raft_consensus.cc:493] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.246364 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.246682 18334 raft_consensus.cc:515] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.247380 18349 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.247951 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
W20260317 16:17:09.248131 18175 leader_election.cc:343] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.248685 18285 raft_consensus.cc:2393] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:09.248809 18334 leader_election.cc:290] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.235664 18355 raft_consensus.cc:493] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.249344 18327 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.249830 18327 ts_tablet_manager.cc:1403] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.008s
I20260317 16:17:09.249671 18355 raft_consensus.cc:515] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.250419 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.251080 18136 raft_consensus.cc:2468] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:09.251783 18355 leader_election.cc:290] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.252580 18327 raft_consensus.cc:359] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.253422 18327 raft_consensus.cc:385] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.253697 18327 raft_consensus.cc:740] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.253753 18178 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.254089 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:09.254329 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.255031 18212 raft_consensus.cc:2393] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:09.255156 18285 raft_consensus.cc:2468] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.255435 18355 raft_consensus.cc:2749] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.256371 18178 leader_election.cc:304] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:09.254360 18327 consensus_queue.cc:260] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.257439 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.257802 18355 raft_consensus.cc:2804] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.258051 18136 raft_consensus.cc:2468] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.258359 18355 raft_consensus.cc:493] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.259214 18327 ts_tablet_manager.cc:1434] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260317 16:17:09.256498 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.259265 18355 raft_consensus.cc:3060] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.260987 18327 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.261049 18249 leader_election.cc:304] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.262521 18334 raft_consensus.cc:2804] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.262934 18334 raft_consensus.cc:493] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.243918 18330 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.252079 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.263259 18334 raft_consensus.cc:3060] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.264767 18330 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.265710 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
W20260317 16:17:09.265925 18249 leader_election.cc:343] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.266386 18249 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.267936 18349 raft_consensus.cc:2749] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.269126 18328 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.268947 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.270287 18328 ts_tablet_manager.cc:1403] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.003s
W20260317 16:17:09.270934 18103 leader_election.cc:343] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.276046 18328 raft_consensus.cc:359] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.277205 18328 raft_consensus.cc:385] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.277487 18328 raft_consensus.cc:740] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.278585 18328 consensus_queue.cc:260] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.279084 18327 tablet_bootstrap.cc:654] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.280932 18349 consensus_queue.cc:1048] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:09.282181 18326 tablet_bootstrap.cc:492] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.282927 18326 ts_tablet_manager.cc:1403] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.080s user 0.022s sys 0.002s
I20260317 16:17:09.245374 18333 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.289059 18326 raft_consensus.cc:359] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.290117 18326 raft_consensus.cc:385] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.290508 18326 raft_consensus.cc:740] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.292237 18353 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.291262 18326 consensus_queue.cc:260] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.284044 18334 raft_consensus.cc:515] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.289664 18379 raft_consensus.cc:493] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.294736 18326 ts_tablet_manager.cc:1434] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20260317 16:17:09.295609 18212 raft_consensus.cc:2393] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:09.298200 18326 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.299403 18102 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:09.301277 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:09.303622 18328 ts_tablet_manager.cc:1434] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.033s user 0.011s sys 0.007s
I20260317 16:17:09.301322 18330 raft_consensus.cc:2749] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.304316 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.304889 18137 raft_consensus.cc:3060] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.304934 18328 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.315526 18137 raft_consensus.cc:2468] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:09.316846 18334 leader_election.cc:290] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.330650 18249 leader_election.cc:304] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:09.322364 18379 raft_consensus.cc:515] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.324910 18326 tablet_bootstrap.cc:654] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.326401 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.333442 18136 raft_consensus.cc:2393] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 94edf207909c4f8990bfe29ed1a87a6d in this term.
I20260317 16:17:09.327060 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.327699 18353 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.323303 18328 tablet_bootstrap.cc:654] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.337608 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:09.336056 18178 leader_election.cc:343] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.338596 18136 raft_consensus.cc:2468] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.339617 18330 raft_consensus.cc:493] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.340298 18330 raft_consensus.cc:515] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.345073 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.345721 18328 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.346190 18328 ts_tablet_manager.cc:1403] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.000s
W20260317 16:17:09.346884 18178 leader_election.cc:343] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.348758 18330 leader_election.cc:290] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.346236 18175 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.349049 18328 raft_consensus.cc:359] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.350443 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ded5b9676242a18139f88dc63af2b4" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.350044 18328 raft_consensus.cc:385] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.351684 18328 raft_consensus.cc:740] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
W20260317 16:17:09.352159 18103 leader_election.cc:343] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.348436 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ded5b9676242a18139f88dc63af2b4" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.352696 18381 raft_consensus.cc:2749] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.353629 18381 raft_consensus.cc:493] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.354384 18379 leader_election.cc:290] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.354246 18381 raft_consensus.cc:515] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.358093 18381 leader_election.cc:290] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:09.359643 18102 leader_election.cc:343] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.360355 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a9551ad4c24868805c90f4fa899ff0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.361387 18285 raft_consensus.cc:2468] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.360357 18102 leader_election.cc:304] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:09.360855 18175 leader_election.cc:304] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:09.364895 18178 leader_election.cc:304] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:09.366073 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a9551ad4c24868805c90f4fa899ff0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.366357 18330 raft_consensus.cc:2749] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.367812 18379 raft_consensus.cc:2804] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.367940 18381 raft_consensus.cc:2804] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.368520 18379 raft_consensus.cc:493] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.369654 18379 raft_consensus.cc:3060] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:09.372447 18175 leader_election.cc:343] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.368546 18381 raft_consensus.cc:493] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.378436 18355 raft_consensus.cc:515] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.379737 18330 raft_consensus.cc:493] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.380275 18211 raft_consensus.cc:2393] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:09.380468 18330 raft_consensus.cc:515] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.385871 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:09.387920 18102 leader_election.cc:343] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.388368 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.388811 18381 raft_consensus.cc:3060] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.389550 18136 raft_consensus.cc:2393] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:09.392002 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:09.393931 18103 leader_election.cc:343] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.394666 18103 leader_election.cc:304] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:09.397693 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.398859 18285 raft_consensus.cc:2393] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:09.400632 18178 leader_election.cc:304] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.401795 18353 raft_consensus.cc:2749] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.402524 18355 leader_election.cc:290] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.398597 18330 leader_election.cc:290] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.403945 18330 raft_consensus.cc:2749] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.405140 18388 raft_consensus.cc:2804] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.414630 18388 raft_consensus.cc:697] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:09.421408 18353 consensus_queue.cc:1048] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:09.352382 18328 consensus_queue.cc:260] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.426292 18328 ts_tablet_manager.cc:1434] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.080s user 0.007s sys 0.001s
I20260317 16:17:09.428324 18327 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.428810 18327 ts_tablet_manager.cc:1403] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.168s user 0.044s sys 0.051s
I20260317 16:17:09.430919 18388 consensus_queue.cc:237] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.431442 18327 raft_consensus.cc:359] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.433465 18327 raft_consensus.cc:385] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.433753 18327 raft_consensus.cc:740] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.433910 18328 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.440081 18379 raft_consensus.cc:515] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.441776 18327 consensus_queue.cc:260] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.442866 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.443418 18136 raft_consensus.cc:3060] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.444271 18326 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.445151 18326 ts_tablet_manager.cc:1403] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.147s user 0.042s sys 0.053s
I20260317 16:17:09.448159 18333 consensus_queue.cc:1048] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:09.444314 18327 ts_tablet_manager.cc:1434] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260317 16:17:09.451306 18136 raft_consensus.cc:2468] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.451797 18327 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.452610 18175 leader_election.cc:304] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.454087 18389 raft_consensus.cc:493] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.454528 18333 raft_consensus.cc:2804] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.454656 18389 raft_consensus.cc:515] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.456815 18333 raft_consensus.cc:697] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:09.458201 18333 consensus_queue.cc:237] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.462841 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.463585 18211 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:09.465340 18102 leader_election.cc:304] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.466298 18330 raft_consensus.cc:2804] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.438211 18381 raft_consensus.cc:515] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.466681 18330 raft_consensus.cc:493] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.467079 18330 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.469270 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a9551ad4c24868805c90f4fa899ff0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.448048 18326 raft_consensus.cc:359] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.470234 18326 raft_consensus.cc:385] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.470556 18326 raft_consensus.cc:740] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.470669 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
W20260317 16:17:09.470873 18175 leader_election.cc:343] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.471493 18285 raft_consensus.cc:3060] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.471346 18326 consensus_queue.cc:260] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.476950 18330 raft_consensus.cc:515] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.477805 18326 ts_tablet_manager.cc:1434] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260317 16:17:09.481125 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.486925 18286 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:09.487898 18330 leader_election.cc:290] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.478513 18389 leader_election.cc:290] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.483136 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a9551ad4c24868805c90f4fa899ff0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.484241 18328 tablet_bootstrap.cc:654] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.488799 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:09.489018 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.489416 18211 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.489574 18286 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.484479 18379 leader_election.cc:290] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.495733 18285 raft_consensus.cc:2468] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.485103 18327 tablet_bootstrap.cc:654] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.488900 18284 raft_consensus.cc:3060] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.487412 18326 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.484820 18381 leader_election.cc:290] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.506964 18211 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:09.508790 18286 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:09.509501 18034 catalog_manager.cc:5671] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:09.522959 18333 raft_consensus.cc:493] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.510288 18103 leader_election.cc:304] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
W20260317 16:17:09.567252 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 57059.7us
I20260317 16:17:09.567376 18389 raft_consensus.cc:2804] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.568115 18389 raft_consensus.cc:697] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:09.569437 18389 consensus_queue.cc:237] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.556540 18333 raft_consensus.cc:515] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.572260 18328 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.572772 18328 ts_tablet_manager.cc:1403] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.139s user 0.014s sys 0.003s
I20260317 16:17:09.583474 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.584292 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.585543 18286 raft_consensus.cc:2468] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
W20260317 16:17:09.587337 18175 leader_election.cc:343] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.587369 18328 raft_consensus.cc:359] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.588186 18328 raft_consensus.cc:385] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:17:09.586618 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:09.588501 18328 raft_consensus.cc:740] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.589044 18178 leader_election.cc:304] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.589311 18328 consensus_queue.cc:260] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.590050 18284 raft_consensus.cc:2468] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.590421 18379 raft_consensus.cc:2804] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.590929 18379 raft_consensus.cc:493] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.591300 18379 raft_consensus.cc:3060] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.591432 18178 leader_election.cc:304] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.591914 18328 ts_tablet_manager.cc:1434] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20260317 16:17:09.593869 18328 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.593773 18355 raft_consensus.cc:2804] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.596405 18355 raft_consensus.cc:697] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:09.599458 18353 raft_consensus.cc:493] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.585817 18333 leader_election.cc:290] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.600258 18353 raft_consensus.cc:515] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.604523 18353 leader_election.cc:290] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.597460 18355 consensus_queue.cc:237] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.604527 18379 raft_consensus.cc:515] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.606220 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.609541 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:09.612483 18175 leader_election.cc:343] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:09.620415 18178 leader_election.cc:343] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.621230 18178 leader_election.cc:304] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.623310 18353 raft_consensus.cc:2749] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.624377 18326 tablet_bootstrap.cc:654] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.642597 18379 leader_election.cc:290] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.642959 18034 catalog_manager.cc:5671] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:09.643368 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
W20260317 16:17:09.646377 18175 leader_election.cc:343] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.654698 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.658510 18388 raft_consensus.cc:493] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.659104 18284 raft_consensus.cc:3060] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.659376 18388 raft_consensus.cc:515] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.664615 18327 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.665304 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:09.667163 18249 leader_election.cc:343] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.668617 18388 leader_election.cc:290] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.665305 18327 ts_tablet_manager.cc:1403] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.214s user 0.017s sys 0.005s
I20260317 16:17:09.671218 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:09.680394 18251 leader_election.cc:343] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.683019 18251 leader_election.cc:304] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.686669 18388 raft_consensus.cc:2749] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.683139 18379 raft_consensus.cc:493] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.687229 18328 tablet_bootstrap.cc:654] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.687482 18379 raft_consensus.cc:515] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.685622 18326 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.682183 18327 raft_consensus.cc:359] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.689086 18326 ts_tablet_manager.cc:1403] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.202s user 0.037s sys 0.047s
I20260317 16:17:09.689419 18327 raft_consensus.cc:385] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.689747 18327 raft_consensus.cc:740] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.688915 18033 catalog_manager.cc:5671] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:09.690598 18327 consensus_queue.cc:260] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.691052 18379 leader_election.cc:290] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.691444 18034 catalog_manager.cc:5671] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:09.694309 18327 ts_tablet_manager.cc:1434] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.023s user 0.010s sys 0.005s
I20260317 16:17:09.696314 18327 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.700078 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.703044 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.704715 18327 tablet_bootstrap.cc:654] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:09.704726 18175 leader_election.cc:343] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.705899 18326 raft_consensus.cc:359] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:09.705739 18178 leader_election.cc:343] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.706589 18326 raft_consensus.cc:385] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.710875 18326 raft_consensus.cc:740] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.711747 18326 consensus_queue.cc:260] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.714336 18326 ts_tablet_manager.cc:1434] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.025s user 0.002s sys 0.003s
I20260317 16:17:09.715390 18326 tablet_bootstrap.cc:492] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.710345 18178 leader_election.cc:304] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.720587 18379 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:09.727519 18379 raft_consensus.cc:2749] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.729912 18284 raft_consensus.cc:2468] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.731614 18178 leader_election.cc:304] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.733897 18379 raft_consensus.cc:2804] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.734400 18379 raft_consensus.cc:697] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:09.742033 18379 consensus_queue.cc:237] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.749771 18355 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:09.760793 18326 tablet_bootstrap.cc:654] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.778594 18328 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.779670 18328 ts_tablet_manager.cc:1403] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.186s user 0.048s sys 0.033s
I20260317 16:17:09.783613 18328 raft_consensus.cc:359] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.784318 18328 raft_consensus.cc:385] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.784587 18328 raft_consensus.cc:740] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.785252 18328 consensus_queue.cc:260] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.787878 18328 ts_tablet_manager.cc:1434] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260317 16:17:09.788841 18328 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.789510 18034 catalog_manager.cc:5671] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:09.795244 18327 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.796443 18327 ts_tablet_manager.cc:1403] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.100s user 0.033s sys 0.024s
I20260317 16:17:09.798549 18328 tablet_bootstrap.cc:654] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.804821 18326 tablet_bootstrap.cc:492] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.805505 18326 ts_tablet_manager.cc:1403] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.023s
I20260317 16:17:09.802186 18327 raft_consensus.cc:359] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.807304 18327 raft_consensus.cc:385] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.807611 18327 raft_consensus.cc:740] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.808334 18327 consensus_queue.cc:260] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.810602 18327 ts_tablet_manager.cc:1434] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260317 16:17:09.811038 18326 raft_consensus.cc:359] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.811730 18327 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.811956 18326 raft_consensus.cc:385] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.814131 18326 raft_consensus.cc:740] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.814743 18326 consensus_queue.cc:260] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.819099 18326 ts_tablet_manager.cc:1434] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20260317 16:17:09.820458 18326 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.827134 18328 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.827636 18328 ts_tablet_manager.cc:1403] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.004s
I20260317 16:17:09.830179 18328 raft_consensus.cc:359] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.831001 18328 raft_consensus.cc:385] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.831276 18328 raft_consensus.cc:740] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.831893 18328 consensus_queue.cc:260] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.839548 18327 tablet_bootstrap.cc:654] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.842172 18326 tablet_bootstrap.cc:654] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.844063 18328 ts_tablet_manager.cc:1434] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260317 16:17:09.845085 18328 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.850356 18328 tablet_bootstrap.cc:654] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.854664 18389 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.855464 18389 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.862802 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.863629 18211 raft_consensus.cc:2393] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:09.866437 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.867264 18284 raft_consensus.cc:2393] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:09.868649 18103 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:09.869520 18330 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.870657 18326 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.871127 18326 ts_tablet_manager.cc:1403] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.011s
I20260317 16:17:09.873386 18326 raft_consensus.cc:359] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.873844 18326 raft_consensus.cc:385] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.874631 18326 raft_consensus.cc:740] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.869561 18389 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.876796 18326 consensus_queue.cc:260] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.880831 18327 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.881930 18327 ts_tablet_manager.cc:1403] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.070s user 0.020s sys 0.007s
I20260317 16:17:09.883757 18330 raft_consensus.cc:2749] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.884857 18328 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.885372 18328 ts_tablet_manager.cc:1403] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.016s
I20260317 16:17:09.885950 18326 ts_tablet_manager.cc:1434] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20260317 16:17:09.886325 18327 raft_consensus.cc:359] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.887146 18326 tablet_bootstrap.cc:492] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.887109 18327 raft_consensus.cc:385] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.887722 18327 raft_consensus.cc:740] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.888345 18328 raft_consensus.cc:359] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.889019 18328 raft_consensus.cc:385] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.888506 18327 consensus_queue.cc:260] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.889293 18328 raft_consensus.cc:740] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.889947 18328 consensus_queue.cc:260] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.894469 18326 tablet_bootstrap.cc:654] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.897166 18327 ts_tablet_manager.cc:1434] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260317 16:17:09.897284 18328 ts_tablet_manager.cc:1434] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260317 16:17:09.898434 18327 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.898670 18328 tablet_bootstrap.cc:492] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:09.903097 18388 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.903645 18388 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.905855 18327 tablet_bootstrap.cc:654] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.905969 18388 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.906859 18328 tablet_bootstrap.cc:654] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.907292 18326 tablet_bootstrap.cc:492] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.907754 18326 ts_tablet_manager.cc:1403] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20260317 16:17:09.907691 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.908638 18137 raft_consensus.cc:2393] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 94edf207909c4f8990bfe29ed1a87a6d in this term.
I20260317 16:17:09.910959 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.911139 18326 raft_consensus.cc:359] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.912104 18326 raft_consensus.cc:385] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.912189 18211 raft_consensus.cc:2393] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:09.912513 18326 raft_consensus.cc:740] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.913411 18326 consensus_queue.cc:260] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.916258 18251 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.916723 18326 ts_tablet_manager.cc:1434] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260317 16:17:09.918008 18328 tablet_bootstrap.cc:492] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:09.918444 18328 ts_tablet_manager.cc:1403] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.010s
I20260317 16:17:09.919198 18388 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.918488 18326 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.921137 18328 raft_consensus.cc:359] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.921774 18328 raft_consensus.cc:385] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.922032 18328 raft_consensus.cc:740] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.923539 18349 raft_consensus.cc:493] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.924008 18349 raft_consensus.cc:515] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.924235 18327 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:09.929553 18327 ts_tablet_manager.cc:1403] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.009s
I20260317 16:17:09.929003 18326 tablet_bootstrap.cc:654] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.932386 18327 raft_consensus.cc:359] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.933485 18327 raft_consensus.cc:385] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.933961 18327 raft_consensus.cc:740] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.935021 18327 consensus_queue.cc:260] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.938565 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ded5b9676242a18139f88dc63af2b4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:09.939360 18137 raft_consensus.cc:2468] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:09.941413 18249 leader_election.cc:304] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.943192 18379 raft_consensus.cc:493] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.943915 18379 raft_consensus.cc:515] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.944833 18388 raft_consensus.cc:2749] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:09.946312 18388 raft_consensus.cc:2804] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.946899 18388 raft_consensus.cc:493] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.947382 18388 raft_consensus.cc:3060] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.948067 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:09.949832 18175 leader_election.cc:343] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:09.952486 18379 leader_election.cc:290] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.952739 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.953426 18284 raft_consensus.cc:2468] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:09.956840 18388 raft_consensus.cc:515] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.959103 18178 leader_election.cc:304] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:09.960322 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ded5b9676242a18139f88dc63af2b4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.960510 18388 leader_election.cc:290] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:09.960708 18327 ts_tablet_manager.cc:1434] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.031s user 0.012s sys 0.012s
I20260317 16:17:09.961004 18211 raft_consensus.cc:2468] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:09.961344 18379 raft_consensus.cc:2804] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.961659 18327 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:09.961736 18379 raft_consensus.cc:493] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.961881 18389 raft_consensus.cc:493] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:09.962222 18379 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.962356 18389 raft_consensus.cc:515] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.964362 18389 leader_election.cc:290] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.964835 18326 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:09.966971 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:09.967626 18284 raft_consensus.cc:2468] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:09.968686 18327 tablet_bootstrap.cc:654] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:09.968900 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:09.969848 18211 raft_consensus.cc:2468] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:09.970408 18326 ts_tablet_manager.cc:1403] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.018s
I20260317 16:17:09.970173 18379 raft_consensus.cc:515] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.971365 18102 leader_election.cc:304] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:09.972229 18379 leader_election.cc:290] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:09.973369 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:09.974205 18284 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.974953 18396 raft_consensus.cc:2804] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:09.975333 18396 raft_consensus.cc:493] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:09.974155 18326 raft_consensus.cc:359] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.975641 18396 raft_consensus.cc:3060] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.976286 18326 raft_consensus.cc:385] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:09.976941 18326 raft_consensus.cc:740] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:09.977725 18326 consensus_queue.cc:260] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.975644 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ded5b9676242a18139f88dc63af2b4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:09.979480 18211 raft_consensus.cc:3060] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:09.980396 18326 ts_tablet_manager.cc:1434] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260317 16:17:09.982713 18284 raft_consensus.cc:2468] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:09.984501 18178 leader_election.cc:304] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:09.987250 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:09.989230 18379 raft_consensus.cc:2804] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:09.989888 18379 raft_consensus.cc:697] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:09.990851 18379 consensus_queue.cc:237] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.991375 18326 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:09.998934 18326 tablet_bootstrap.cc:654] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.004783 18326 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.005254 18326 ts_tablet_manager.cc:1403] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20260317 16:17:10.006876 18349 leader_election.cc:290] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.008536 18326 raft_consensus.cc:359] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:09.922943 18328 consensus_queue.cc:260] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.009433 18326 raft_consensus.cc:385] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.010167 18326 raft_consensus.cc:740] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.010355 18396 raft_consensus.cc:515] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:10.002741 18175 leader_election.cc:343] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.011914 18326 consensus_queue.cc:260] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.013460 18328 ts_tablet_manager.cc:1434] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.095s user 0.006s sys 0.001s
I20260317 16:17:10.013518 18330 raft_consensus.cc:493] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.014089 18330 raft_consensus.cc:515] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.015161 18328 tablet_bootstrap.cc:492] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
W20260317 16:17:10.010914 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 69606.7us
I20260317 16:17:10.016363 18379 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.017478 18330 leader_election.cc:290] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.019156 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.019919 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:10.020489 18212 raft_consensus.cc:3060] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:10.020630 18102 leader_election.cc:343] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.020674 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:10.019192 18396 leader_election.cc:290] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.021375 18285 raft_consensus.cc:3060] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.023277 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.024358 18326 ts_tablet_manager.cc:1434] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.019s user 0.008s sys 0.002s
W20260317 16:17:10.025712 18103 leader_election.cc:343] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.026343 18103 leader_election.cc:304] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.031793 18326 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.037436 18327 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.037261 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ded5b9676242a18139f88dc63af2b4" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:10.037609 18211 raft_consensus.cc:2468] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:10.037823 18137 raft_consensus.cc:3060] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.036619 18033 catalog_manager.cc:5671] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:10.037878 18327 ts_tablet_manager.cc:1403] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.076s user 0.021s sys 0.029s
I20260317 16:17:10.041940 18327 raft_consensus.cc:359] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.043025 18327 raft_consensus.cc:385] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.043334 18327 raft_consensus.cc:740] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.044457 18326 tablet_bootstrap.cc:654] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.044232 18327 consensus_queue.cc:260] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.044991 18388 raft_consensus.cc:493] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.031632 18418 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260317 16:17:10.045437 18388 raft_consensus.cc:515] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.027998 18396 raft_consensus.cc:2749] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.053804 18388 leader_election.cc:290] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.055981 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.059293 18396 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.049294 18418 raft_consensus.cc:493] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.060894 18327 ts_tablet_manager.cc:1434] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260317 16:17:10.061252 18418 raft_consensus.cc:515] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.062500 18327 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.063395 18379 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.068145 18251 leader_election.cc:304] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
W20260317 16:17:10.068506 18249 leader_election.cc:343] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.070739 18285 raft_consensus.cc:2468] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:10.071835 18379 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.075024 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.078893 18212 raft_consensus.cc:2468] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:10.078140 18349 raft_consensus.cc:2804] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.072270 18396 raft_consensus.cc:493] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.083047 18349 raft_consensus.cc:697] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:10.085553 18102 leader_election.cc:304] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:10.088758 18417 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.091254 18389 raft_consensus.cc:2804] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.091362 18349 consensus_queue.cc:237] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.078527 18379 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.076288 18418 leader_election.cc:290] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.092088 18389 raft_consensus.cc:697] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:10.105315 18327 tablet_bootstrap.cc:654] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:10.108630 18251 leader_election.cc:343] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.109300 18251 leader_election.cc:304] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:10.110155 18417 raft_consensus.cc:2749] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.110972 18326 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.113301 18326 ts_tablet_manager.cc:1403] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.082s user 0.023s sys 0.025s
I20260317 16:17:10.116948 18328 tablet_bootstrap.cc:654] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.086547 18396 raft_consensus.cc:515] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.125589 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7a4647163b461f87a0a249491e566c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.123129 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7a4647163b461f87a0a249491e566c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.133677 18355 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.134171 18379 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.137113 18396 leader_election.cc:290] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.144299 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.157307 18353 raft_consensus.cc:493] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.158684 18353 raft_consensus.cc:515] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.156432 18326 raft_consensus.cc:359] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.127935 18389 consensus_queue.cc:237] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.163811 18326 raft_consensus.cc:385] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.164144 18326 raft_consensus.cc:740] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.164649 18353 leader_election.cc:290] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.165352 18326 consensus_queue.cc:260] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:10.167438 18251 leader_election.cc:343] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.178248 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a9551ad4c24868805c90f4fa899ff0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.181537 18137 raft_consensus.cc:2468] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:10.192198 18132 raft_consensus.cc:3060] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.193929 18396 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.199210 18285 raft_consensus.cc:2393] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:10.191516 18135 raft_consensus.cc:2468] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
W20260317 16:17:10.188652 18178 leader_election.cc:343] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.201598 18249 leader_election.cc:304] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.189731 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a9551ad4c24868805c90f4fa899ff0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.202615 18418 raft_consensus.cc:2804] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.182493 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.203006 18418 raft_consensus.cc:493] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.203341 18418 raft_consensus.cc:3060] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.204494 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "701d2829bc6a450a94470d7f852ecdc8" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:10.204005 18175 leader_election.cc:343] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.205297 18175 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:10.206990 18102 leader_election.cc:304] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.207706 18355 raft_consensus.cc:2749] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.207542 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "701d2829bc6a450a94470d7f852ecdc8" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.214936 18355 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.224299 18418 raft_consensus.cc:515] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:10.225546 18175 leader_election.cc:343] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:10.230952 18178 leader_election.cc:343] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.231441 18178 leader_election.cc:304] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:10.231757 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7a4647163b461f87a0a249491e566c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:10.231930 18418 leader_election.cc:290] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.232372 18136 raft_consensus.cc:3060] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.235038 18328 tablet_bootstrap.cc:492] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:10.232476 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7a4647163b461f87a0a249491e566c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:10.235567 18328 ts_tablet_manager.cc:1403] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.221s user 0.010s sys 0.006s
I20260317 16:17:10.241493 18136 raft_consensus.cc:2468] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
W20260317 16:17:10.242609 18251 leader_election.cc:343] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.243340 18249 leader_election.cc:304] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.244640 18417 raft_consensus.cc:2804] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.245019 18427 raft_consensus.cc:2749] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.245263 18417 raft_consensus.cc:697] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:10.246474 18417 consensus_queue.cc:237] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.227411 18326 ts_tablet_manager.cc:1434] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.114s user 0.008s sys 0.002s
I20260317 16:17:10.248193 18326 tablet_bootstrap.cc:492] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.232731 18353 raft_consensus.cc:2749] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.258410 18326 tablet_bootstrap.cc:654] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.260207 18327 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.260764 18427 raft_consensus.cc:493] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.261442 18427 raft_consensus.cc:515] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.238261 18328 raft_consensus.cc:359] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.263010 18328 raft_consensus.cc:385] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.260875 18327 ts_tablet_manager.cc:1403] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.199s user 0.017s sys 0.009s
I20260317 16:17:10.265861 18355 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.269961 18327 raft_consensus.cc:359] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.272596 18327 raft_consensus.cc:385] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.271364 18389 raft_consensus.cc:493] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.273173 18327 raft_consensus.cc:740] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.274109 18327 consensus_queue.cc:260] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.276396 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.274855 18427 leader_election.cc:290] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.278113 18327 ts_tablet_manager.cc:1434] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20260317 16:17:10.278604 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.279292 18327 tablet_bootstrap.cc:492] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.280026 18102 leader_election.cc:304] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.263332 18328 raft_consensus.cc:740] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.281519 18328 consensus_queue.cc:260] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.273464 18389 raft_consensus.cc:515] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.284111 18434 raft_consensus.cc:2749] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.285499 18328 ts_tablet_manager.cc:1434] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260317 16:17:10.286247 18136 raft_consensus.cc:3060] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.287277 18134 raft_consensus.cc:3060] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.291064 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.290122 18034 catalog_manager.cc:5671] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:10.298409 18389 leader_election.cc:290] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.303406 18333 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.304236 18328 tablet_bootstrap.cc:492] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:10.304996 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f14079a4ea34cbda92828ca274134f4" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.308604 18035 catalog_manager.cc:5671] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:10.316069 18103 leader_election.cc:304] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.320801 18327 tablet_bootstrap.cc:654] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.327926 18328 tablet_bootstrap.cc:654] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.329234 18326 tablet_bootstrap.cc:492] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.329641 18326 ts_tablet_manager.cc:1403] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.001s
I20260317 16:17:10.344182 18326 raft_consensus.cc:359] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.345139 18326 raft_consensus.cc:385] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.345474 18326 raft_consensus.cc:740] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.346247 18326 consensus_queue.cc:260] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.354802 18427 raft_consensus.cc:2749] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.354147 18034 catalog_manager.cc:5671] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:10.355502 18326 ts_tablet_manager.cc:1434] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.026s user 0.007s sys 0.003s
I20260317 16:17:10.357040 18326 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.368678 18327 tablet_bootstrap.cc:492] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.369277 18327 ts_tablet_manager.cc:1403] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.005s
I20260317 16:17:10.369971 18326 tablet_bootstrap.cc:654] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.371987 18327 raft_consensus.cc:359] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.372733 18327 raft_consensus.cc:385] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.373019 18327 raft_consensus.cc:740] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.373663 18327 consensus_queue.cc:260] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.376812 18327 ts_tablet_manager.cc:1434] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260317 16:17:10.377916 18327 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.388900 18327 tablet_bootstrap.cc:654] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.390538 18417 raft_consensus.cc:493] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.391052 18417 raft_consensus.cc:515] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.395424 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.397032 18417 leader_election.cc:290] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:10.398038 18251 leader_election.cc:343] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.400533 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:10.401770 18249 leader_election.cc:343] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.402209 18327 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.402189 18249 leader_election.cc:304] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:10.402740 18327 ts_tablet_manager.cc:1403] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.002s
I20260317 16:17:10.403421 18417 raft_consensus.cc:2749] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.405234 18327 raft_consensus.cc:359] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.405897 18327 raft_consensus.cc:385] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.406160 18327 raft_consensus.cc:740] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.406793 18327 consensus_queue.cc:260] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.408915 18327 ts_tablet_manager.cc:1434] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260317 16:17:10.410140 18327 tablet_bootstrap.cc:492] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.416635 18327 tablet_bootstrap.cc:654] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.430208 18136 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.447068 18379 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.447808 18328 tablet_bootstrap.cc:492] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:10.448292 18328 ts_tablet_manager.cc:1403] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.144s user 0.056s sys 0.061s
I20260317 16:17:10.449710 18326 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.450438 18326 ts_tablet_manager.cc:1403] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.094s user 0.026s sys 0.039s
I20260317 16:17:10.450893 18328 raft_consensus.cc:359] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.451598 18328 raft_consensus.cc:385] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.451879 18328 raft_consensus.cc:740] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.452530 18328 consensus_queue.cc:260] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.454111 18327 tablet_bootstrap.cc:492] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.454629 18327 ts_tablet_manager.cc:1403] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.021s
I20260317 16:17:10.454612 18326 raft_consensus.cc:359] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.456686 18328 ts_tablet_manager.cc:1434] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260317 16:17:10.456861 18326 raft_consensus.cc:385] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.457413 18326 raft_consensus.cc:740] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.457922 18328 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:10.458398 18326 consensus_queue.cc:260] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.462239 18326 ts_tablet_manager.cc:1434] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260317 16:17:10.463598 18326 tablet_bootstrap.cc:492] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.466579 18328 tablet_bootstrap.cc:654] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.470940 18326 tablet_bootstrap.cc:654] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.474665 18327 raft_consensus.cc:359] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.475836 18327 raft_consensus.cc:385] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.476373 18327 raft_consensus.cc:740] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.477617 18327 consensus_queue.cc:260] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.479278 18328 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:10.480178 18328 ts_tablet_manager.cc:1403] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.004s
I20260317 16:17:10.483476 18328 raft_consensus.cc:359] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.484205 18328 raft_consensus.cc:385] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.484493 18328 raft_consensus.cc:740] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.485484 18326 tablet_bootstrap.cc:492] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.485111 18328 consensus_queue.cc:260] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.485966 18326 ts_tablet_manager.cc:1403] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.006s
I20260317 16:17:10.487373 18328 ts_tablet_manager.cc:1434] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260317 16:17:10.488528 18328 tablet_bootstrap.cc:492] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:10.488680 18326 raft_consensus.cc:359] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.489383 18326 raft_consensus.cc:385] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.489660 18326 raft_consensus.cc:740] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.490273 18326 consensus_queue.cc:260] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.492242 18326 ts_tablet_manager.cc:1434] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260317 16:17:10.497620 18328 tablet_bootstrap.cc:654] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.499203 18327 ts_tablet_manager.cc:1434] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260317 16:17:10.501442 18327 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.504171 18326 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.521134 18379 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.521648 18326 tablet_bootstrap.cc:654] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.524422 18418 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.524933 18418 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.526057 18327 tablet_bootstrap.cc:654] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.527280 18418 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.529320 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.529938 18211 raft_consensus.cc:2468] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:10.530423 18328 tablet_bootstrap.cc:492] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:10.531040 18328 ts_tablet_manager.cc:1403] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.015s
I20260317 16:17:10.531087 18251 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:10.531785 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.532085 18418 raft_consensus.cc:2804] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.532431 18418 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.532877 18418 raft_consensus.cc:3060] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:10.534441 18249 leader_election.cc:343] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.535593 18328 raft_consensus.cc:359] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.536587 18328 raft_consensus.cc:385] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.536935 18328 raft_consensus.cc:740] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.537819 18328 consensus_queue.cc:260] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.553102 18328 ts_tablet_manager.cc:1434] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20260317 16:17:10.553134 18327 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.553730 18327 ts_tablet_manager.cc:1403] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.006s
I20260317 16:17:10.553679 18418 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.554322 18328 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:10.554356 18417 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.557090 18327 raft_consensus.cc:359] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.557915 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:10.558073 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:10.561103 18327 raft_consensus.cc:385] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.561933 18327 raft_consensus.cc:740] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.561425 18211 raft_consensus.cc:3060] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.563410 18327 consensus_queue.cc:260] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.566576 18327 ts_tablet_manager.cc:1434] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.012s user 0.000s sys 0.006s
I20260317 16:17:10.567888 18327 tablet_bootstrap.cc:492] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.562886 18418 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:10.576462 18249 leader_election.cc:343] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.577329 18328 tablet_bootstrap.cc:654] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.580442 18427 raft_consensus.cc:493] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.583722 18427 raft_consensus.cc:515] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.585403 18418 raft_consensus.cc:493] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.586031 18427 leader_election.cc:290] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.587189 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "701d2829bc6a450a94470d7f852ecdc8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.588625 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "701d2829bc6a450a94470d7f852ecdc8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.589584 18212 raft_consensus.cc:2468] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:10.590862 18327 tablet_bootstrap.cc:654] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:10.589366 18103 leader_election.cc:343] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:10.597576 18102 leader_election.cc:304] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:10.591825 18418 raft_consensus.cc:515] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:10.599103 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:10.599037 18427 raft_consensus.cc:2804] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.599696 18427 raft_consensus.cc:493] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.600145 18427 raft_consensus.cc:3060] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.601562 18326 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.601300 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.602037 18326 ts_tablet_manager.cc:1403] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.098s user 0.018s sys 0.023s
I20260317 16:17:10.602059 18136 raft_consensus.cc:2468] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:10.598048 18211 raft_consensus.cc:2468] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:10.609483 18249 leader_election.cc:304] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:10.610641 18349 raft_consensus.cc:2804] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.611076 18349 raft_consensus.cc:493] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.611384 18349 raft_consensus.cc:3060] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.612011 18326 raft_consensus.cc:359] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.612679 18326 raft_consensus.cc:385] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.612970 18326 raft_consensus.cc:740] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.613584 18326 consensus_queue.cc:260] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.616132 18326 ts_tablet_manager.cc:1434] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20260317 16:17:10.617168 18326 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.618655 18349 raft_consensus.cc:515] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.623910 18251 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:10.635298 18417 raft_consensus.cc:2804] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.636034 18417 raft_consensus.cc:697] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:10.644263 18389 raft_consensus.cc:493] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.645102 18389 raft_consensus.cc:515] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.645841 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:10.646533 18136 raft_consensus.cc:3060] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.647801 18389 leader_election.cc:290] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.648957 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.649636 18284 raft_consensus.cc:2468] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:10.651082 18379 raft_consensus.cc:493] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.651252 18103 leader_election.cc:304] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:10.651628 18379 raft_consensus.cc:515] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.651881 18418 leader_election.cc:290] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.654278 18349 leader_election.cc:290] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.652391 18389 raft_consensus.cc:2804] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.676062 18389 raft_consensus.cc:493] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.676573 18389 raft_consensus.cc:3060] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.678426 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.671202 18326 tablet_bootstrap.cc:654] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.637495 18417 consensus_queue.cc:237] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.675444 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.679824 18379 leader_election.cc:290] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:10.689806 18251 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 66307.1us
I20260317 16:17:10.689832 18330 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:10.685313 18211 raft_consensus.cc:2468] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:10.694897 18327 tablet_bootstrap.cc:492] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:10.695379 18327 ts_tablet_manager.cc:1403] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.128s user 0.052s sys 0.043s
I20260317 16:17:10.695658 18328 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:10.696141 18328 ts_tablet_manager.cc:1403] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.142s user 0.020s sys 0.022s
I20260317 16:17:10.701444 18327 raft_consensus.cc:359] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.702663 18327 raft_consensus.cc:385] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.703070 18327 raft_consensus.cc:740] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.699467 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.704129 18327 consensus_queue.cc:260] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.704787 18211 raft_consensus.cc:2468] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:10.707438 18209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:10.712831 18209 raft_consensus.cc:3060] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.713538 18327 ts_tablet_manager.cc:1434] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260317 16:17:10.714810 18327 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:10.717312 18434 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.708899 18136 raft_consensus.cc:2468] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:10.707669 18427 raft_consensus.cc:515] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.742125 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.742892 18285 raft_consensus.cc:2468] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:10.743041 18427 leader_election.cc:290] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.729796 18249 leader_election.cc:304] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:10.744402 18388 raft_consensus.cc:2804] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.745658 18388 raft_consensus.cc:697] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:10.745831 18178 leader_election.cc:304] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:10.746938 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "701d2829bc6a450a94470d7f852ecdc8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:10.747406 18379 raft_consensus.cc:2804] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.747705 18211 raft_consensus.cc:3060] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.747784 18379 raft_consensus.cc:493] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.748211 18379 raft_consensus.cc:3060] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.745676 18457 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260317 16:17:10.752194 18389 raft_consensus.cc:515] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.752779 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "701d2829bc6a450a94470d7f852ecdc8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:10.753364 18286 raft_consensus.cc:3060] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.754319 18328 raft_consensus.cc:359] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.755151 18328 raft_consensus.cc:385] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.755416 18328 raft_consensus.cc:740] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.756201 18328 consensus_queue.cc:260] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.757932 18134 raft_consensus.cc:2393] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 94edf207909c4f8990bfe29ed1a87a6d in this term.
I20260317 16:17:10.758062 18328 ts_tablet_manager.cc:1434] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20260317 16:17:10.759482 18328 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:10.765226 18457 raft_consensus.cc:493] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.783035 18457 raft_consensus.cc:515] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.785445 18209 raft_consensus.cc:2468] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:10.790144 18286 raft_consensus.cc:2468] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:10.746999 18388 consensus_queue.cc:237] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.811363 18379 raft_consensus.cc:515] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.864259 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:10.865121 18134 raft_consensus.cc:2393] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 94edf207909c4f8990bfe29ed1a87a6d in this term.
W20260317 16:17:10.865830 18251 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 72789.1us
I20260317 16:17:10.864837 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:10.873124 18326 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:10.875546 18326 ts_tablet_manager.cc:1403] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.259s user 0.015s sys 0.006s
I20260317 16:17:10.872229 18103 leader_election.cc:304] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:10.884698 18210 raft_consensus.cc:2393] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:10.867959 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.862674 18355 raft_consensus.cc:493] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.866140 18457 leader_election.cc:290] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.885440 18389 leader_election.cc:290] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.888459 18427 raft_consensus.cc:2804] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.886932 18379 leader_election.cc:290] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.895141 18136 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:10.894891 18355 raft_consensus.cc:515] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.897611 18427 raft_consensus.cc:697] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:10.904968 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:10.908813 18249 leader_election.cc:304] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
W20260317 16:17:10.912503 18459 log.cc:927] Time spent T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260317 16:17:10.910094 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:10.921655 18326 raft_consensus.cc:359] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.922319 18326 raft_consensus.cc:385] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:10.922581 18326 raft_consensus.cc:740] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:10.915616 18285 raft_consensus.cc:3060] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.925352 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:10.921561 18457 raft_consensus.cc:2804] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.926335 18133 raft_consensus.cc:2468] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:10.931020 18457 raft_consensus.cc:493] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.924358 18326 consensus_queue.cc:260] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.931515 18457 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.923383 18427 consensus_queue.cc:237] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.933599 18326 ts_tablet_manager.cc:1434] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.058s user 0.005s sys 0.003s
I20260317 16:17:10.929045 18103 leader_election.cc:304] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:10.933682 18285 raft_consensus.cc:2468] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:10.934337 18434 raft_consensus.cc:2749] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:10.927645 18327 tablet_bootstrap.cc:654] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:10.935459 18326 tablet_bootstrap.cc:492] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:10.935806 18417 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:10.937788 18175 leader_election.cc:304] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:10.922506 18355 leader_election.cc:290] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.940083 18463 raft_consensus.cc:2804] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:10.940589 18463 raft_consensus.cc:493] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:10.941028 18463 raft_consensus.cc:3060] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.964095 18212 raft_consensus.cc:3060] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:10.947556 18178 leader_election.cc:304] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:10.964095 18209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:10.972121 18355 raft_consensus.cc:2804] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:10.966310 18211 raft_consensus.cc:2468] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:10.975790 18333 raft_consensus.cc:493] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:10.976287 18333 raft_consensus.cc:515] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.978165 18333 leader_election.cc:290] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:10.965253 18457 raft_consensus.cc:515] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.978616 18209 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:10.986003 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.987632 18035 catalog_manager.cc:5671] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:10.972635 18355 raft_consensus.cc:697] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:10.991429 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:10.995119 18462 log.cc:927] Time spent T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260317 16:17:10.995602 18355 consensus_queue.cc:237] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:10.993348 18034 catalog_manager.cc:5671] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:10.997170 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:10.999271 18284 raft_consensus.cc:2468] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
W20260317 16:17:10.996696 18178 leader_election.cc:343] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.001161 18034 catalog_manager.cc:5671] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
W20260317 16:17:11.003774 18175 leader_election.cc:343] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.007045 18464 raft_consensus.cc:493] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.007308 18328 tablet_bootstrap.cc:654] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.007545 18464 raft_consensus.cc:515] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.009163 18178 leader_election.cc:304] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.010090 18333 raft_consensus.cc:2804] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 16:17:11.010437 18460 log.cc:927] Time spent T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260317 16:17:11.010433 18333 raft_consensus.cc:493] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:11.011054 18333 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.012056 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7a4647163b461f87a0a249491e566c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.013504 18457 leader_election.cc:290] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:10.997489 18463 raft_consensus.cc:515] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.026240 18427 raft_consensus.cc:493] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:17:11.030125 18465 log.cc:927] Time spent T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260317 16:17:11.032241 18457 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.033068 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7a4647163b461f87a0a249491e566c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.033572 18326 tablet_bootstrap.cc:654] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.044059 18333 raft_consensus.cc:515] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.049878 18178 leader_election.cc:304] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.044240 18427 raft_consensus.cc:515] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.053427 18427 leader_election.cc:290] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.025923 18464 leader_election.cc:290] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.026594 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.056056 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.056715 18136 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.036034 18463 leader_election.cc:290] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.036906 18330 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.057283 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:11.057904 18285 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.057734 18330 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:11.059521 18103 leader_election.cc:343] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.060422 18330 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.062490 18209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:11.054425 18333 leader_election.cc:290] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.063400 18209 raft_consensus.cc:2393] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:11.037441 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:11.038425 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.064345 18134 raft_consensus.cc:3060] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.039134 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
W20260317 16:17:11.065192 18178 leader_election.cc:343] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.054313 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:11.065305 18212 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.065455 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.052098 18355 raft_consensus.cc:2749] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:17:11.067065 18102 leader_election.cc:343] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.068849 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.069046 18103 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:11.069420 18102 leader_election.cc:304] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:11.070309 18330 raft_consensus.cc:3060] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.070556 18427 raft_consensus.cc:2749] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:17:11.075726 18175 leader_election.cc:343] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.080370 18327 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:11.080920 18327 ts_tablet_manager.cc:1403] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.366s user 0.031s sys 0.018s
I20260317 16:17:11.088582 18328 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:11.089124 18328 ts_tablet_manager.cc:1403] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.330s user 0.033s sys 0.028s
I20260317 16:17:11.093665 18285 raft_consensus.cc:2468] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.093600 18136 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:11.094976 18178 leader_election.cc:304] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.095422 18249 leader_election.cc:304] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:11.096109 18355 raft_consensus.cc:2804] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:11.096480 18418 raft_consensus.cc:2804] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:11.096601 18355 raft_consensus.cc:697] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:11.096990 18418 raft_consensus.cc:697] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:11.097461 18355 consensus_queue.cc:237] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.097765 18418 consensus_queue.cc:237] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.098603 18134 raft_consensus.cc:2468] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.103338 18175 leader_election.cc:304] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:11.104502 18463 raft_consensus.cc:2804] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:11.107273 18463 raft_consensus.cc:697] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:11.108105 18463 consensus_queue.cc:237] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.113302 18333 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.113962 18333 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.114634 18327 raft_consensus.cc:359] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.115432 18327 raft_consensus.cc:385] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.115713 18327 raft_consensus.cc:740] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.116770 18327 consensus_queue.cc:260] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.118700 18333 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.120038 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.121004 18285 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.121094 18327 ts_tablet_manager.cc:1434] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.040s user 0.006s sys 0.003s
I20260317 16:17:11.122066 18178 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:11.122345 18327 tablet_bootstrap.cc:492] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:11.122911 18463 raft_consensus.cc:2804] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:17:11.122676 18328 raft_consensus.cc:359] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.120144 18330 raft_consensus.cc:2749] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:11.124516 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.123508 18328 raft_consensus.cc:385] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.123485 18463 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:11.125815 18328 raft_consensus.cc:740] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.125924 18463 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:11.125859 18349 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:11.118049 18212 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:11.126616 18328 consensus_queue.cc:260] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.128981 18328 ts_tablet_manager.cc:1434] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20260317 16:17:11.129885 18328 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:11.132115 18327 tablet_bootstrap.cc:654] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.125530 18136 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.162185 18328 tablet_bootstrap.cc:654] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.163163 18418 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.159121 18032 catalog_manager.cc:5671] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:11.159067 18463 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.167974 18326 tablet_bootstrap.cc:492] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:11.168502 18326 ts_tablet_manager.cc:1403] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.233s user 0.025s sys 0.013s
I20260317 16:17:11.168416 18463 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.179176 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.179834 18134 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:11.183075 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:11.183677 18285 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:11.191107 18326 raft_consensus.cc:359] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.191867 18326 raft_consensus.cc:385] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.192155 18326 raft_consensus.cc:740] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.192772 18326 consensus_queue.cc:260] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.202980 18034 catalog_manager.cc:5671] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:11.208848 18134 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:11.211587 18175 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:11.212414 18285 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:11.212617 18463 raft_consensus.cc:2804] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:11.213450 18463 raft_consensus.cc:697] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:11.214411 18463 consensus_queue.cc:237] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.225814 18326 ts_tablet_manager.cc:1434] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.057s user 0.005s sys 0.001s
I20260317 16:17:11.227010 18326 tablet_bootstrap.cc:492] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:11.282090 18035 catalog_manager.cc:5671] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:11.285080 18035 catalog_manager.cc:5671] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:11.299144 18355 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.299716 18355 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.302443 18327 tablet_bootstrap.cc:492] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:11.302955 18327 ts_tablet_manager.cc:1403] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.181s user 0.013s sys 0.006s
I20260317 16:17:11.307509 18355 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.309805 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.309136 18326 tablet_bootstrap.cc:654] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.316926 18327 raft_consensus.cc:359] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.322327 18327 raft_consensus.cc:385] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.322698 18327 raft_consensus.cc:740] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.325043 18328 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:11.325426 18328 ts_tablet_manager.cc:1403] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.196s user 0.012s sys 0.006s
I20260317 16:17:11.309919 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.328928 18285 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.330525 18178 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:11.332695 18333 raft_consensus.cc:2804] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:17:11.333077 18333 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:11.333405 18333 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:11.326952 18134 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.336436 18328 raft_consensus.cc:359] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.337124 18328 raft_consensus.cc:385] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.337411 18328 raft_consensus.cc:740] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.338021 18328 consensus_queue.cc:260] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.339675 18431 consensus_queue.cc:1048] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:11.339728 18328 ts_tablet_manager.cc:1434] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260317 16:17:11.348678 18328 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:11.358053 18333 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.327659 18327 consensus_queue.cc:260] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.362201 18327 ts_tablet_manager.cc:1434] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.059s user 0.007s sys 0.002s
I20260317 16:17:11.363181 18327 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:11.364313 18434 consensus_queue.cc:1048] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.367794 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.368398 18132 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:11.368973 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:11.370994 18285 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:11.373756 18333 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.378238 18328 tablet_bootstrap.cc:654] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.379419 18327 tablet_bootstrap.cc:654] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.382629 18326 tablet_bootstrap.cc:492] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:11.422092 18333 raft_consensus.cc:493] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.424116 18333 raft_consensus.cc:515] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.432253 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.435772 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.436961 18284 raft_consensus.cc:2468] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:11.438629 18178 leader_election.cc:304] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
W20260317 16:17:11.440080 18175 leader_election.cc:343] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.441366 18463 raft_consensus.cc:2804] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:11.441838 18463 raft_consensus.cc:493] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:11.442322 18463 raft_consensus.cc:3060] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.443145 18349 raft_consensus.cc:493] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.443892 18349 raft_consensus.cc:515] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.449869 18285 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:11.450881 18333 leader_election.cc:290] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.453855 18132 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:11.454046 18464 consensus_queue.cc:1048] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:11.466594 18349 leader_election.cc:290] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:11.462441 18326 ts_tablet_manager.cc:1403] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.236s user 0.030s sys 0.028s
I20260317 16:17:11.467208 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.482945 18327 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:11.483647 18327 ts_tablet_manager.cc:1403] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.121s user 0.030s sys 0.013s
I20260317 16:17:11.479748 18463 raft_consensus.cc:515] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.467700 18175 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:11.485965 18326 raft_consensus.cc:359] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.475323 18328 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:11.487499 18328 ts_tablet_manager.cc:1403] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.012s
I20260317 16:17:11.487597 18463 leader_election.cc:290] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.488044 18327 raft_consensus.cc:359] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.488754 18327 raft_consensus.cc:385] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.488850 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:11.487501 18333 raft_consensus.cc:2804] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:11.489414 18284 raft_consensus.cc:3060] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.489665 18333 raft_consensus.cc:697] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:11.490976 18333 consensus_queue.cc:237] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.491379 18328 raft_consensus.cc:359] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.473565 18032 catalog_manager.cc:5671] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 2, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:11.492280 18328 raft_consensus.cc:385] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.492707 18328 raft_consensus.cc:740] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.493497 18328 consensus_queue.cc:260] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.493911 18134 raft_consensus.cc:2468] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:11.493925 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.495838 18249 leader_election.cc:304] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:11.489082 18327 raft_consensus.cc:740] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.496822 18349 raft_consensus.cc:2804] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:11.496172 18326 raft_consensus.cc:385] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.497292 18349 raft_consensus.cc:493] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:11.497740 18349 raft_consensus.cc:3060] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.496922 18327 consensus_queue.cc:260] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.497633 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:11.501504 18212 raft_consensus.cc:2468] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:11.503680 18327 ts_tablet_manager.cc:1434] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.020s user 0.005s sys 0.004s
I20260317 16:17:11.505384 18327 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:11.497471 18326 raft_consensus.cc:740] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.505856 18463 consensus_queue.cc:1048] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.506206 18326 consensus_queue.cc:260] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.507872 18284 raft_consensus.cc:2468] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:11.500229 18328 ts_tablet_manager.cc:1434] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20260317 16:17:11.511015 18328 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
W20260317 16:17:11.503750 18175 leader_election.cc:343] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.515084 18326 ts_tablet_manager.cc:1434] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.033s user 0.005s sys 0.004s
I20260317 16:17:11.520747 18388 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260317 16:17:11.528216 18326 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:11.519608 18178 leader_election.cc:304] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.532259 18388 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.532958 18349 raft_consensus.cc:515] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.535115 18349 leader_election.cc:290] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:11.536085 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:11.536988 18136 raft_consensus.cc:3060] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.541064 18333 raft_consensus.cc:2804] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:11.541620 18333 raft_consensus.cc:697] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:11.541867 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:11.542505 18207 raft_consensus.cc:3060] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.542665 18333 consensus_queue.cc:237] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.551467 18330 raft_consensus.cc:493] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.552094 18330 raft_consensus.cc:515] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.565428 18330 leader_election.cc:290] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.571657 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:11.576774 18102 leader_election.cc:343] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.580111 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.574287 18328 tablet_bootstrap.cc:654] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.558712 18388 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
W20260317 16:17:11.594738 18103 leader_election.cc:343] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.595419 18103 leader_election.cc:304] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:11.596993 18330 raft_consensus.cc:2749] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:11.615236 18463 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.615837 18463 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.616887 18136 raft_consensus.cc:2468] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:11.617965 18463 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.623546 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.626067 18249 leader_election.cc:304] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:11.627523 18417 raft_consensus.cc:2804] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:11.631202 18355 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:17:11.637130 18175 leader_election.cc:343] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.648476 18417 raft_consensus.cc:697] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:11.662238 18207 raft_consensus.cc:2468] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:11.670397 18417 consensus_queue.cc:237] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.693239 18418 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:11.668888 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.700733 18327 tablet_bootstrap.cc:654] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.702025 18326 tablet_bootstrap.cc:654] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.689335 18034 catalog_manager.cc:5671] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 2, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:11.719041 18175 consensus_peers.cc:597] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:11.742118 18178 leader_election.cc:343] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.744463 18328 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:11.745164 18328 ts_tablet_manager.cc:1403] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.235s user 0.026s sys 0.024s
I20260317 16:17:11.746019 18284 raft_consensus.cc:3060] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.757072 18132 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:17:11.757756 18333 consensus_queue.cc:1048] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.758899 18463 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.759042 18283 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:17:11.759524 18330 raft_consensus.cc:493] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.760035 18330 raft_consensus.cc:515] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.762346 18330 leader_election.cc:290] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.774173 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:11.784484 18328 raft_consensus.cc:359] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.785284 18328 raft_consensus.cc:385] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.785630 18328 raft_consensus.cc:740] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.792975 18328 consensus_queue.cc:260] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:11.801807 18102 leader_election.cc:343] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.742589 18178 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.808145 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:11.810318 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260317 16:17:11.819024 18103 leader_election.cc:343] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.819613 18103 leader_election.cc:304] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:11.852581 18035 catalog_manager.cc:5671] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:11.854139 18388 raft_consensus.cc:493] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.854885 18388 raft_consensus.cc:515] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.855665 18501 raft_consensus.cc:2749] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:17:11.859714 18178 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 117619us
I20260317 16:17:11.868961 18327 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:11.869911 18327 ts_tablet_manager.cc:1403] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.365s user 0.021s sys 0.016s
I20260317 16:17:11.874327 18328 ts_tablet_manager.cc:1434] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.129s user 0.010s sys 0.005s
I20260317 16:17:11.879046 18328 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:11.875849 18330 raft_consensus.cc:2749] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:17:11.876080 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 57125.2us
I20260317 16:17:11.880748 18333 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 16:17:11.880748 18327 raft_consensus.cc:359] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.887943 18327 raft_consensus.cc:385] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.888269 18327 raft_consensus.cc:740] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.889092 18327 consensus_queue.cc:260] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.894518 18327 ts_tablet_manager.cc:1434] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.024s user 0.008s sys 0.001s
I20260317 16:17:11.896363 18327 tablet_bootstrap.cc:492] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:11.901914 18501 consensus_queue.cc:1048] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:11.912317 18328 tablet_bootstrap.cc:654] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.918170 18388 leader_election.cc:290] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:11.903574 18327 tablet_bootstrap.cc:654] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:11.937767 18464 raft_consensus.cc:493] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.938313 18464 raft_consensus.cc:515] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.940598 18464 leader_election.cc:290] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:11.944559 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:11.946157 18175 leader_election.cc:343] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.947715 18034 catalog_manager.cc:5671] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:11.946221 18134 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:17:11.952220 18464 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:11.904145 18326 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:11.957908 18326 ts_tablet_manager.cc:1403] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.430s user 0.030s sys 0.012s
I20260317 16:17:11.960632 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:11.964867 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbff882fd6aa4aa0a32d72d1674abc76" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:11.966676 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:11.969394 18285 raft_consensus.cc:2468] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:11.972682 18326 raft_consensus.cc:359] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.975003 18251 leader_election.cc:304] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.979036 18326 raft_consensus.cc:385] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.979622 18326 raft_consensus.cc:740] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.976161 18328 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:11.975775 18178 leader_election.cc:304] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:11.980834 18328 ts_tablet_manager.cc:1403] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.102s user 0.037s sys 0.017s
I20260317 16:17:11.980852 18457 raft_consensus.cc:493] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:11.981416 18464 raft_consensus.cc:2804] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:11.981508 18457 raft_consensus.cc:515] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.981833 18464 raft_consensus.cc:493] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:11.979647 18506 raft_consensus.cc:2749] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:11.982601 18464 raft_consensus.cc:3060] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:11.983480 18328 raft_consensus.cc:359] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.984266 18328 raft_consensus.cc:385] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:11.985474 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:11.987040 18249 leader_election.cc:343] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:11.984266 18457 leader_election.cc:290] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:11.984640 18328 raft_consensus.cc:740] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:11.989737 18327 tablet_bootstrap.cc:492] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:11.989131 18328 consensus_queue.cc:260] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.990329 18327 ts_tablet_manager.cc:1403] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.004s
I20260317 16:17:11.991452 18464 raft_consensus.cc:515] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:11.993480 18327 raft_consensus.cc:359] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:11.996071 18175 consensus_peers.cc:597] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:12.004122 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:11.997447 18328 ts_tablet_manager.cc:1434] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260317 16:17:11.996264 18326 consensus_queue.cc:260] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.005940 18328 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
W20260317 16:17:12.005998 18251 leader_election.cc:343] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.006851 18326 ts_tablet_manager.cc:1434] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260317 16:17:12.006668 18251 leader_election.cc:304] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.007210 18327 raft_consensus.cc:385] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.007846 18326 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.008090 18457 raft_consensus.cc:2749] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.009680 18327 raft_consensus.cc:740] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.010957 18327 consensus_queue.cc:260] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.015148 18327 ts_tablet_manager.cc:1434] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.024s user 0.002s sys 0.005s
I20260317 16:17:12.016134 18464 leader_election.cc:290] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.016389 18327 tablet_bootstrap.cc:492] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:12.016832 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.020108 18328 tablet_bootstrap.cc:654] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:12.020363 18175 leader_election.cc:343] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.015147 18285 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:17:12.023407 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:12.024012 18284 raft_consensus.cc:3060] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.024577 18327 tablet_bootstrap.cc:654] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.033842 18327 tablet_bootstrap.cc:492] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.033628 18464 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:12.034283 18327 ts_tablet_manager.cc:1403] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260317 16:17:12.038409 18284 raft_consensus.cc:2468] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:12.044052 18327 raft_consensus.cc:359] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.044777 18327 raft_consensus.cc:385] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.045035 18327 raft_consensus.cc:740] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.045070 18178 leader_election.cc:304] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.045753 18327 consensus_queue.cc:260] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.046352 18464 raft_consensus.cc:2804] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.046864 18464 raft_consensus.cc:697] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:12.048568 18328 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.047854 18464 consensus_queue.cc:237] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.049034 18328 ts_tablet_manager.cc:1403] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.003s
I20260317 16:17:12.051673 18328 raft_consensus.cc:359] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.052203 18327 ts_tablet_manager.cc:1434] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20260317 16:17:12.052376 18328 raft_consensus.cc:385] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.052693 18328 raft_consensus.cc:740] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.053570 18327 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:12.053390 18328 consensus_queue.cc:260] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.055524 18328 ts_tablet_manager.cc:1434] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260317 16:17:12.044052 18326 tablet_bootstrap.cc:654] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.058118 18328 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.065941 18328 tablet_bootstrap.cc:654] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.069576 18327 tablet_bootstrap.cc:654] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.082315 18328 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.082813 18328 ts_tablet_manager.cc:1403] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s
I20260317 16:17:12.085137 18328 raft_consensus.cc:359] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.085776 18328 raft_consensus.cc:385] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.086036 18328 raft_consensus.cc:740] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.086866 18328 consensus_queue.cc:260] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.089807 18326 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.090294 18326 ts_tablet_manager.cc:1403] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.003s
I20260317 16:17:12.089692 18033 catalog_manager.cc:5671] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:12.090998 18327 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.091421 18327 ts_tablet_manager.cc:1403] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.004s
I20260317 16:17:12.092684 18326 raft_consensus.cc:359] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.093466 18326 raft_consensus.cc:385] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.094122 18326 raft_consensus.cc:740] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.093787 18327 raft_consensus.cc:359] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.094584 18327 raft_consensus.cc:385] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.094993 18327 raft_consensus.cc:740] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.096088 18327 consensus_queue.cc:260] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.097575 18464 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:12.098260 18327 ts_tablet_manager.cc:1434] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260317 16:17:12.098845 18328 ts_tablet_manager.cc:1434] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260317 16:17:12.099887 18328 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.100273 18326 consensus_queue.cc:260] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.102414 18326 ts_tablet_manager.cc:1434] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.012s user 0.001s sys 0.005s
I20260317 16:17:12.103868 18326 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.106050 18328 tablet_bootstrap.cc:654] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.112005 18326 tablet_bootstrap.cc:654] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.120981 18327 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:12.130180 18327 tablet_bootstrap.cc:654] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.136932 18328 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.137622 18328 ts_tablet_manager.cc:1403] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.004s
I20260317 16:17:12.144604 18328 raft_consensus.cc:359] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.145512 18328 raft_consensus.cc:385] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.145844 18328 raft_consensus.cc:740] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.147555 18328 consensus_queue.cc:260] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.149513 18327 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.150627 18327 ts_tablet_manager.cc:1403] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.000s
I20260317 16:17:12.153189 18327 raft_consensus.cc:359] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.153923 18327 raft_consensus.cc:385] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.154198 18327 raft_consensus.cc:740] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.149857 18326 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.154999 18326 ts_tablet_manager.cc:1403] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.008s
I20260317 16:17:12.154917 18327 consensus_queue.cc:260] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.157277 18327 ts_tablet_manager.cc:1434] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260317 16:17:12.150161 18328 ts_tablet_manager.cc:1434] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260317 16:17:12.157671 18326 raft_consensus.cc:359] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.158336 18326 raft_consensus.cc:385] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.158463 18328 tablet_bootstrap.cc:492] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.158648 18326 raft_consensus.cc:740] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.158861 18327 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:12.159335 18326 consensus_queue.cc:260] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.161697 18326 ts_tablet_manager.cc:1434] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260317 16:17:12.162595 18514 raft_consensus.cc:493] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.163215 18326 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.163126 18514 raft_consensus.cc:515] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.165261 18328 tablet_bootstrap.cc:654] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.166929 18327 tablet_bootstrap.cc:654] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.166903 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.168991 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.170307 18326 tablet_bootstrap.cc:654] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.172709 18103 leader_election.cc:304] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:12.173986 18434 raft_consensus.cc:2749] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.175603 18134 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.179502 18514 leader_election.cc:290] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.185190 18499 raft_consensus.cc:493] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.185742 18499 raft_consensus.cc:515] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.187922 18499 leader_election.cc:290] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.191145 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.191947 18132 raft_consensus.cc:2468] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:12.193262 18175 leader_election.cc:304] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:12.194864 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.196666 18328 tablet_bootstrap.cc:492] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.196630 18333 raft_consensus.cc:2804] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.197232 18328 ts_tablet_manager.cc:1403] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.009s
I20260317 16:17:12.197149 18333 raft_consensus.cc:493] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:17:12.197721 18178 leader_election.cc:343] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.198086 18333 raft_consensus.cc:3060] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.200881 18328 raft_consensus.cc:359] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.201637 18328 raft_consensus.cc:385] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.201922 18328 raft_consensus.cc:740] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.197057 18506 raft_consensus.cc:493] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.203362 18506 raft_consensus.cc:515] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.204408 18328 consensus_queue.cc:260] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.205521 18506 leader_election.cc:290] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.206866 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.207150 18327 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.207718 18327 ts_tablet_manager.cc:1403] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.049s user 0.027s sys 0.015s
I20260317 16:17:12.209190 18328 ts_tablet_manager.cc:1434] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260317 16:17:12.206511 18506 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260317 16:17:12.210300 18328 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.209484 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.210652 18327 raft_consensus.cc:359] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.211174 18134 raft_consensus.cc:2468] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
W20260317 16:17:12.211259 18251 leader_election.cc:343] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.211459 18327 raft_consensus.cc:385] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.211925 18327 raft_consensus.cc:740] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.212394 18249 leader_election.cc:304] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:12.213507 18506 raft_consensus.cc:2804] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.213860 18506 raft_consensus.cc:493] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.214212 18506 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.217140 18328 tablet_bootstrap.cc:654] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.219166 18333 raft_consensus.cc:515] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.220461 18327 consensus_queue.cc:260] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.222744 18333 leader_election.cc:290] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.224584 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:12.224305 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.225917 18132 raft_consensus.cc:3060] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:12.226302 18178 leader_election.cc:343] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.233078 18328 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.233583 18328 ts_tablet_manager.cc:1403] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.009s
I20260317 16:17:12.225634 18327 ts_tablet_manager.cc:1434] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260317 16:17:12.233560 18132 raft_consensus.cc:2468] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:12.234946 18506 raft_consensus.cc:515] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.235864 18175 leader_election.cc:304] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:12.236337 18327 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:12.237787 18506 leader_election.cc:290] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.237923 18326 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.240159 18326 ts_tablet_manager.cc:1403] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.013s
I20260317 16:17:12.239092 18328 raft_consensus.cc:359] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.240980 18328 raft_consensus.cc:385] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.241287 18328 raft_consensus.cc:740] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.241950 18328 consensus_queue.cc:260] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.242743 18326 raft_consensus.cc:359] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.243405 18326 raft_consensus.cc:385] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.243317 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.243790 18326 raft_consensus.cc:740] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.244092 18134 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.244423 18328 ts_tablet_manager.cc:1434] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260317 16:17:12.244522 18326 consensus_queue.cc:260] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.245779 18328 tablet_bootstrap.cc:492] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.247051 18326 ts_tablet_manager.cc:1434] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260317 16:17:12.247684 18333 raft_consensus.cc:2804] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.248330 18333 raft_consensus.cc:697] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:12.248610 18326 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.249228 18333 consensus_queue.cc:237] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.249822 18327 tablet_bootstrap.cc:654] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.252502 18328 tablet_bootstrap.cc:654] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.253753 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
W20260317 16:17:12.255007 18251 leader_election.cc:343] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.276619 18326 tablet_bootstrap.cc:654] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.283883 18506 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:12.289099 18134 raft_consensus.cc:2468] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:12.291260 18249 leader_election.cc:304] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:12.292654 18457 raft_consensus.cc:2804] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.293157 18457 raft_consensus.cc:697] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:12.293998 18457 consensus_queue.cc:237] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.298924 18326 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.299566 18326 ts_tablet_manager.cc:1403] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.024s
I20260317 16:17:12.302946 18326 raft_consensus.cc:359] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.303665 18326 raft_consensus.cc:385] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.303941 18326 raft_consensus.cc:740] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.304648 18326 consensus_queue.cc:260] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.306986 18326 ts_tablet_manager.cc:1434] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260317 16:17:12.309370 18328 tablet_bootstrap.cc:492] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.309854 18328 ts_tablet_manager.cc:1403] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.005s
I20260317 16:17:12.306134 18032 catalog_manager.cc:5671] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:12.312952 18328 raft_consensus.cc:359] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.313567 18328 raft_consensus.cc:385] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.313810 18328 raft_consensus.cc:740] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.314302 18500 raft_consensus.cc:493] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.314406 18328 consensus_queue.cc:260] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.315318 18326 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.314913 18500 raft_consensus.cc:515] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.316900 18328 ts_tablet_manager.cc:1434] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.007s user 0.004s sys 0.005s
I20260317 16:17:12.317041 18500 leader_election.cc:290] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.317842 18328 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.320469 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.321604 18326 tablet_bootstrap.cc:654] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.322428 18514 raft_consensus.cc:493] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:17:12.322594 18178 leader_election.cc:343] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.322953 18514 raft_consensus.cc:515] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.324865 18514 leader_election.cc:290] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.326445 18328 tablet_bootstrap.cc:654] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.326915 18333 raft_consensus.cc:493] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.327379 18333 raft_consensus.cc:515] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.328377 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.330329 18333 leader_election.cc:290] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.332643 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.333263 18134 raft_consensus.cc:2468] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:12.334825 18175 leader_election.cc:304] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:12.340215 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.340878 18284 raft_consensus.cc:2468] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:12.342203 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b643cd863d14f36a3b9341312ef4770" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.343618 18103 leader_election.cc:304] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:12.345160 18514 raft_consensus.cc:2749] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.343003 18328 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.346421 18328 ts_tablet_manager.cc:1403] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.008s
I20260317 16:17:12.347048 18500 raft_consensus.cc:2804] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.347467 18500 raft_consensus.cc:493] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.347856 18500 raft_consensus.cc:3060] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.349035 18499 raft_consensus.cc:493] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.349654 18499 raft_consensus.cc:515] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.352617 18499 leader_election.cc:290] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.355811 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:12.359508 18175 leader_election.cc:343] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.362010 18457 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.361984 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.363042 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.362896 18457 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.372426 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.372529 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.377278 18211 raft_consensus.cc:2468] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:12.370380 18457 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:12.365553 18178 leader_election.cc:343] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.378885 18178 leader_election.cc:304] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.380175 18499 raft_consensus.cc:2749] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:17:12.378087 18249 leader_election.cc:343] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.367275 18328 raft_consensus.cc:359] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.382252 18251 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.382963 18328 raft_consensus.cc:385] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.383392 18328 raft_consensus.cc:740] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.384025 18457 raft_consensus.cc:2804] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.384544 18457 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.385171 18457 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.384462 18328 consensus_queue.cc:260] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.388125 18328 ts_tablet_manager.cc:1434] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.041s user 0.005s sys 0.002s
I20260317 16:17:12.389544 18328 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
W20260317 16:17:12.402523 18175 leader_election.cc:343] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.403152 18175 leader_election.cc:304] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.405637 18499 raft_consensus.cc:2749] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.410384 18328 tablet_bootstrap.cc:654] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.411739 18033 catalog_manager.cc:5671] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:12.411777 18499 raft_consensus.cc:493] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.418396 18499 raft_consensus.cc:515] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.420379 18499 leader_election.cc:290] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.421886 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:12.423620 18178 leader_election.cc:343] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.423852 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:12.426088 18175 leader_election.cc:343] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.426535 18175 leader_election.cc:304] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.427561 18457 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.428130 18333 raft_consensus.cc:2749] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.432437 18457 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.432701 18328 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.433447 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.433921 18328 ts_tablet_manager.cc:1403] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.002s
I20260317 16:17:12.419263 18500 raft_consensus.cc:515] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.439230 18500 leader_election.cc:290] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:12.440285 18249 leader_election.cc:343] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.440335 18328 raft_consensus.cc:359] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.441295 18328 raft_consensus.cc:385] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.441592 18328 raft_consensus.cc:740] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.442265 18328 consensus_queue.cc:260] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.444053 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:12.444489 18328 ts_tablet_manager.cc:1434] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:17:12.444882 18285 raft_consensus.cc:3060] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.445976 18328 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.446348 18327 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.446813 18327 ts_tablet_manager.cc:1403] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.211s user 0.036s sys 0.045s
I20260317 16:17:12.449642 18327 raft_consensus.cc:359] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.450469 18327 raft_consensus.cc:385] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.450781 18327 raft_consensus.cc:740] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.451479 18327 consensus_queue.cc:260] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.453874 18328 tablet_bootstrap.cc:654] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.454010 18327 ts_tablet_manager.cc:1434] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260317 16:17:12.457257 18327 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:12.459717 18285 raft_consensus.cc:2468] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:12.461000 18178 leader_election.cc:304] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:12.461305 18328 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.461854 18328 ts_tablet_manager.cc:1403] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.002s
I20260317 16:17:12.461716 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.462462 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:12.463351 18211 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.467842 18326 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.468307 18326 ts_tablet_manager.cc:1403] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.153s user 0.043s sys 0.076s
I20260317 16:17:12.462606 18132 raft_consensus.cc:3060] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.473816 18326 raft_consensus.cc:359] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.474684 18326 raft_consensus.cc:385] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.475040 18326 raft_consensus.cc:740] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.475826 18326 consensus_queue.cc:260] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.474903 18211 raft_consensus.cc:2468] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:12.477717 18132 raft_consensus.cc:2468] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:12.464349 18328 raft_consensus.cc:359] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.467121 18500 raft_consensus.cc:2804] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.479189 18328 raft_consensus.cc:385] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.479388 18500 raft_consensus.cc:697] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:12.479591 18328 raft_consensus.cc:740] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.480471 18328 consensus_queue.cc:260] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.480512 18500 consensus_queue.cc:237] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.481038 18251 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.482882 18328 ts_tablet_manager.cc:1434] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260317 16:17:12.483911 18328 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.467422 18327 tablet_bootstrap.cc:654] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.484477 18506 raft_consensus.cc:2804] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.479274 18326 ts_tablet_manager.cc:1434] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260317 16:17:12.485208 18506 raft_consensus.cc:697] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:12.485810 18326 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.486081 18506 consensus_queue.cc:237] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.489926 18457 raft_consensus.cc:493] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.490710 18328 tablet_bootstrap.cc:654] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.490438 18457 raft_consensus.cc:515] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.497913 18417 raft_consensus.cc:493] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.499924 18417 raft_consensus.cc:515] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.501066 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.501896 18132 raft_consensus.cc:2468] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:12.509562 18326 tablet_bootstrap.cc:654] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.510350 18249 leader_election.cc:304] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:12.511253 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.512024 18132 raft_consensus.cc:2468] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:12.498705 18457 leader_election.cc:290] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.512795 18504 raft_consensus.cc:493] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.505079 18417 leader_election.cc:290] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.505228 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.513486 18504 raft_consensus.cc:515] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.514277 18211 raft_consensus.cc:2468] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:12.511291 18502 raft_consensus.cc:2804] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.509720 18327 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.517787 18327 ts_tablet_manager.cc:1403] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.004s
I20260317 16:17:12.517220 18502 raft_consensus.cc:493] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.518560 18502 raft_consensus.cc:3060] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.518894 18249 leader_election.cc:304] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:12.524814 18506 raft_consensus.cc:2804] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.525288 18506 raft_consensus.cc:493] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.525667 18506 raft_consensus.cc:3060] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.526405 18504 leader_election.cc:290] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.532266 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.532644 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.533319 18208 raft_consensus.cc:2468] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:12.533456 18135 raft_consensus.cc:3060] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.532266 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:12.534113 18249 leader_election.cc:343] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.538439 18032 catalog_manager.cc:5671] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:12.540035 18251 leader_election.cc:304] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.540446 18328 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.540925 18328 ts_tablet_manager.cc:1403] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.014s
I20260317 16:17:12.536309 18211 raft_consensus.cc:2468] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:12.542719 18504 raft_consensus.cc:2804] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.543205 18504 raft_consensus.cc:493] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.543593 18504 raft_consensus.cc:3060] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.543952 18328 raft_consensus.cc:359] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.545568 18328 raft_consensus.cc:385] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.546080 18328 raft_consensus.cc:740] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.533584 18502 raft_consensus.cc:515] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.554848 18333 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:12.555521 18502 leader_election.cc:290] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.544317 18500 raft_consensus.cc:493] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.558293 18327 raft_consensus.cc:359] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.548234 18328 consensus_queue.cc:260] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.559077 18327 raft_consensus.cc:385] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.559773 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:12.558013 18500 raft_consensus.cc:515] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.560346 18211 raft_consensus.cc:3060] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.559383 18327 raft_consensus.cc:740] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.570976 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.572329 18284 raft_consensus.cc:2468] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:12.574627 18178 leader_election.cc:304] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:12.575842 18333 raft_consensus.cc:2804] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.576277 18333 raft_consensus.cc:493] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.576651 18333 raft_consensus.cc:3060] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.576972 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.577584 18132 raft_consensus.cc:3060] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.578665 18032 catalog_manager.cc:5671] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:12.581550 18327 consensus_queue.cc:260] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.586606 18333 raft_consensus.cc:515] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.592847 18514 raft_consensus.cc:493] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.593531 18514 raft_consensus.cc:515] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.596035 18502 raft_consensus.cc:493] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.596545 18502 raft_consensus.cc:515] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.596079 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.598610 18502 leader_election.cc:290] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.603315 18514 leader_election.cc:290] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.604586 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.605916 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.606671 18208 raft_consensus.cc:2468] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:12.608152 18102 leader_election.cc:304] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
W20260317 16:17:12.609457 18103 leader_election.cc:343] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.590214 18500 leader_election.cc:290] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.611158 18514 raft_consensus.cc:2804] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:12.609400 18132 raft_consensus.cc:2468] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:12.613756 18514 raft_consensus.cc:493] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:12.614185 18514 raft_consensus.cc:3060] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.614697 18249 leader_election.cc:304] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
W20260317 16:17:12.617602 18175 leader_election.cc:343] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.619525 18333 leader_election.cc:290] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.619917 18328 ts_tablet_manager.cc:1434] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.079s user 0.007s sys 0.003s
I20260317 16:17:12.611192 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.625152 18327 ts_tablet_manager.cc:1434] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.107s user 0.006s sys 0.004s
I20260317 16:17:12.627296 18328 tablet_bootstrap.cc:492] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.623673 18502 raft_consensus.cc:2804] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.629458 18502 raft_consensus.cc:697] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:12.630017 18327 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
W20260317 16:17:12.634418 18249 leader_election.cc:343] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.636444 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:12.630332 18502 consensus_queue.cc:237] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.639223 18284 raft_consensus.cc:3060] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.644220 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.674263 18211 raft_consensus.cc:2468] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
W20260317 16:17:12.674902 18251 leader_election.cc:343] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.676033 18251 leader_election.cc:304] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.676757 18417 raft_consensus.cc:2749] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.675426 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.680658 18506 raft_consensus.cc:515] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.686196 18326 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.701229 18514 raft_consensus.cc:515] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.715008 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.721817 18134 raft_consensus.cc:3060] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.700716 18506 leader_election.cc:290] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.698189 18504 raft_consensus.cc:515] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.720285 18326 ts_tablet_manager.cc:1403] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.235s user 0.028s sys 0.046s
I20260317 16:17:12.709026 18417 consensus_queue.cc:1048] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260317 16:17:12.686904 18284 raft_consensus.cc:2468] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
W20260317 16:17:12.724772 18175 leader_election.cc:343] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.730733 18504 leader_election.cc:290] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.731132 18326 raft_consensus.cc:359] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.735569 18326 raft_consensus.cc:385] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:17:12.735664 18175 consensus_peers.cc:597] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:12.735984 18326 raft_consensus.cc:740] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.734365 18328 tablet_bootstrap.cc:654] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.737066 18326 consensus_queue.cc:260] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.741202 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:12.741819 18211 raft_consensus.cc:3060] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.762981 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:12.758749 18033 catalog_manager.cc:5671] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
W20260317 16:17:12.765275 18103 leader_election.cc:343] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.753840 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:12.762533 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:12.766386 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:12.766917 18212 raft_consensus.cc:3060] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:12.767722 18249 leader_election.cc:343] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.766986 18208 raft_consensus.cc:3060] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:12.770591 18327 tablet_bootstrap.cc:654] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.761741 18514 leader_election.cc:290] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.772162 18211 raft_consensus.cc:2468] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:12.782317 18326 ts_tablet_manager.cc:1434] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.056s user 0.012s sys 0.006s
I20260317 16:17:12.783438 18326 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.795450 18178 leader_election.cc:304] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.800004 18556 raft_consensus.cc:2804] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.800694 18556 raft_consensus.cc:697] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:12.801882 18556 consensus_queue.cc:237] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.804042 18134 raft_consensus.cc:2468] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:12.806949 18251 leader_election.cc:304] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:12.810611 18506 raft_consensus.cc:2804] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.811264 18506 raft_consensus.cc:697] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:12.812156 18506 consensus_queue.cc:237] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.816473 18557 raft_consensus.cc:493] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.817008 18557 raft_consensus.cc:515] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.819268 18557 leader_election.cc:290] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.809244 18212 raft_consensus.cc:2468] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:12.819856 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.820559 18102 leader_election.cc:304] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
W20260317 16:17:12.821195 18175 leader_election.cc:343] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.821576 18559 raft_consensus.cc:2804] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.822482 18559 raft_consensus.cc:697] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:12.823697 18559 consensus_queue.cc:237] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.827093 18208 raft_consensus.cc:2468] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:12.829617 18327 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:12.830120 18327 ts_tablet_manager.cc:1403] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.201s user 0.009s sys 0.013s
I20260317 16:17:12.834890 18326 tablet_bootstrap.cc:654] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.836356 18327 raft_consensus.cc:359] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.837173 18327 raft_consensus.cc:385] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.837512 18327 raft_consensus.cc:740] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.838752 18327 consensus_queue.cc:260] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.840970 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.836769 18328 tablet_bootstrap.cc:492] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:12.843847 18328 ts_tablet_manager.cc:1403] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.218s user 0.017s sys 0.023s
I20260317 16:17:12.846551 18328 raft_consensus.cc:359] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.847259 18328 raft_consensus.cc:385] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.847520 18328 raft_consensus.cc:740] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.848230 18556 raft_consensus.cc:493] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:17:12.848553 18251 consensus_peers.cc:597] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:12.848637 18556 raft_consensus.cc:515] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.849817 18251 leader_election.cc:304] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.851059 18457 raft_consensus.cc:2804] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:12.851521 18457 raft_consensus.cc:697] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:12.851863 18327 ts_tablet_manager.cc:1434] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.021s user 0.003s sys 0.005s
I20260317 16:17:12.852099 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.852414 18556 leader_election.cc:290] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.853245 18327 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
W20260317 16:17:12.853610 18175 leader_election.cc:343] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.852344 18457 consensus_queue.cc:237] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.856761 18328 consensus_queue.cc:260] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.861605 18328 ts_tablet_manager.cc:1434] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260317 16:17:12.862586 18328 tablet_bootstrap.cc:492] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:12.867538 18034 catalog_manager.cc:5671] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:12.879374 18032 catalog_manager.cc:5671] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:12.890982 18457 raft_consensus.cc:493] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.891534 18457 raft_consensus.cc:515] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.894017 18457 leader_election.cc:290] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:12.900847 18326 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:12.901641 18326 ts_tablet_manager.cc:1403] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.118s user 0.011s sys 0.003s
I20260317 16:17:12.904678 18556 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:12.906692 18326 raft_consensus.cc:359] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.907469 18326 raft_consensus.cc:385] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:12.907788 18326 raft_consensus.cc:740] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:12.908577 18326 consensus_queue.cc:260] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.909031 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:12.909832 18132 raft_consensus.cc:2393] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:12.911084 18326 ts_tablet_manager.cc:1434] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260317 16:17:12.912176 18326 tablet_bootstrap.cc:492] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:12.912387 18557 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:12.914367 18033 catalog_manager.cc:5671] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:12.897742 18251 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 48022.9us
I20260317 16:17:12.927788 18559 raft_consensus.cc:493] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.930166 18559 raft_consensus.cc:515] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:12.931417 18178 consensus_peers.cc:597] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:12.935323 18178 leader_election.cc:343] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.935830 18178 leader_election.cc:304] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.936492 18559 leader_election.cc:290] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.937243 18499 raft_consensus.cc:2749] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.938679 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:12.938679 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.948110 18102 leader_election.cc:304] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:12.949424 18514 raft_consensus.cc:2749] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.950999 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:12.951653 18499 raft_consensus.cc:493] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:12.952093 18499 raft_consensus.cc:515] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:12.952534 18326 tablet_bootstrap.cc:654] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.952548 18328 tablet_bootstrap.cc:654] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.969513 18327 tablet_bootstrap.cc:654] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:12.971086 18034 catalog_manager.cc:5671] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:12.965909 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:12.969434 18178 leader_election.cc:343] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:12.980639 18178 leader_election.cc:304] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:12.985445 18557 raft_consensus.cc:2749] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:12.985808 18556 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:12.991532 18499 leader_election.cc:290] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:12.993263 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:12.995437 18175 leader_election.cc:343] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.000522 18328 tablet_bootstrap.cc:492] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:13.001120 18328 ts_tablet_manager.cc:1403] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.139s user 0.016s sys 0.004s
I20260317 16:17:13.001519 18251 leader_election.cc:304] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.006919 18457 raft_consensus.cc:3060] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.024089 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:13.038072 18249 consensus_peers.cc:597] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:13.042958 18178 leader_election.cc:343] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.043485 18178 leader_election.cc:304] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.044761 18499 raft_consensus.cc:2749] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.045401 18514 raft_consensus.cc:493] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.045884 18514 raft_consensus.cc:515] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.048866 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.046115 18328 raft_consensus.cc:359] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.049553 18208 raft_consensus.cc:2468] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:13.049815 18328 raft_consensus.cc:385] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.050364 18328 raft_consensus.cc:740] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.050952 18102 leader_election.cc:304] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:13.051935 18559 raft_consensus.cc:2804] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:13.052285 18559 raft_consensus.cc:493] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.052611 18559 raft_consensus.cc:3060] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.053690 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:13.051117 18328 consensus_queue.cc:260] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.052665 18514 leader_election.cc:290] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:13.056551 18103 leader_election.cc:343] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.022264 18457 raft_consensus.cc:2749] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.071511 18556 raft_consensus.cc:493] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.072129 18556 raft_consensus.cc:515] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.074535 18556 leader_election.cc:290] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.076234 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:13.078719 18326 tablet_bootstrap.cc:492] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:13.079233 18326 ts_tablet_manager.cc:1403] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.027s
I20260317 16:17:13.080089 18327 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:13.082513 18326 raft_consensus.cc:359] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.083249 18326 raft_consensus.cc:385] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.083523 18326 raft_consensus.cc:740] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.080547 18327 ts_tablet_manager.cc:1403] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.228s user 0.017s sys 0.018s
I20260317 16:17:13.084131 18326 consensus_queue.cc:260] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.084857 18328 ts_tablet_manager.cc:1434] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.082s user 0.006s sys 0.000s
I20260317 16:17:13.086042 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:13.087848 18178 leader_election.cc:304] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.089121 18499 raft_consensus.cc:2749] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.090144 18328 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:13.094470 18327 raft_consensus.cc:359] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.095228 18327 raft_consensus.cc:385] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.095479 18327 raft_consensus.cc:740] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.096122 18327 consensus_queue.cc:260] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.098440 18327 ts_tablet_manager.cc:1434] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260317 16:17:13.099493 18327 tablet_bootstrap.cc:492] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:13.104285 18328 tablet_bootstrap.cc:654] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.107692 18326 ts_tablet_manager.cc:1434] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20260317 16:17:13.108903 18326 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:13.110541 18134 raft_consensus.cc:3060] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.112712 18327 tablet_bootstrap.cc:654] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.097930 18559 raft_consensus.cc:515] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.115793 18326 tablet_bootstrap.cc:654] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.117496 18559 leader_election.cc:290] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.118719 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:13.119663 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:13.119689 18208 raft_consensus.cc:3060] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:13.122429 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:13.127779 18457 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:17:13.129487 18103 leader_election.cc:343] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.145529 18326 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:13.149566 18327 tablet_bootstrap.cc:492] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:13.150244 18327 ts_tablet_manager.cc:1403] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.001s
I20260317 16:17:13.152437 18326 ts_tablet_manager.cc:1403] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.001s
I20260317 16:17:13.154418 18208 raft_consensus.cc:2468] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:13.155401 18326 raft_consensus.cc:359] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.156299 18326 raft_consensus.cc:385] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.155946 18102 leader_election.cc:304] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:13.156606 18326 raft_consensus.cc:740] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.157306 18559 raft_consensus.cc:2804] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.157804 18559 raft_consensus.cc:697] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:13.158625 18559 consensus_queue.cc:237] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.165131 18326 consensus_queue.cc:260] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.170909 18326 ts_tablet_manager.cc:1434] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260317 16:17:13.171973 18326 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:13.187433 18327 raft_consensus.cc:359] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.188146 18327 raft_consensus.cc:385] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.188413 18327 raft_consensus.cc:740] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.189163 18327 consensus_queue.cc:260] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.195062 18328 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:13.195541 18328 ts_tablet_manager.cc:1403] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.020s
I20260317 16:17:13.197960 18328 raft_consensus.cc:359] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.198647 18328 raft_consensus.cc:385] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.198951 18328 raft_consensus.cc:740] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.199862 18327 ts_tablet_manager.cc:1434] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.049s user 0.005s sys 0.005s
I20260317 16:17:13.199572 18328 consensus_queue.cc:260] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.203611 18327 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:13.202946 18326 tablet_bootstrap.cc:654] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.205870 18033 catalog_manager.cc:5671] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:13.211261 18327 tablet_bootstrap.cc:654] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.211946 18328 ts_tablet_manager.cc:1434] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260317 16:17:13.213354 18328 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:13.218849 18504 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:13.221197 18326 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:13.221693 18326 ts_tablet_manager.cc:1403] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.006s
I20260317 16:17:13.224493 18326 raft_consensus.cc:359] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.225178 18326 raft_consensus.cc:385] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.225436 18326 raft_consensus.cc:740] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.226083 18326 consensus_queue.cc:260] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.227119 18556 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:13.228379 18326 ts_tablet_manager.cc:1434] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260317 16:17:13.230213 18327 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:13.230918 18327 ts_tablet_manager.cc:1403] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.009s
I20260317 16:17:13.231808 18328 tablet_bootstrap.cc:654] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.233649 18327 raft_consensus.cc:359] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.234483 18327 raft_consensus.cc:385] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.234687 18326 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:13.234853 18327 raft_consensus.cc:740] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.235626 18327 consensus_queue.cc:260] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.237828 18327 ts_tablet_manager.cc:1434] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260317 16:17:13.238818 18327 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:13.242681 18457 consensus_queue.cc:1048] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:13.244709 18327 tablet_bootstrap.cc:654] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.247900 18328 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:13.248353 18328 ts_tablet_manager.cc:1403] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.000s
I20260317 16:17:13.250798 18328 raft_consensus.cc:359] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.251157 18499 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.251497 18328 raft_consensus.cc:385] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.251804 18328 raft_consensus.cc:740] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.251852 18499 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.252491 18328 consensus_queue.cc:260] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.254074 18326 tablet_bootstrap.cc:654] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.256884 18499 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.258582 18328 ts_tablet_manager.cc:1434] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260317 16:17:13.258456 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:13.260342 18178 leader_election.cc:343] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.261359 18328 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:13.267400 18328 tablet_bootstrap.cc:654] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.268352 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:13.271186 18175 leader_election.cc:343] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.271711 18175 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.274011 18499 raft_consensus.cc:2749] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.274677 18514 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.275393 18514 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.276456 18326 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:13.276921 18326 ts_tablet_manager.cc:1403] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.000s
I20260317 16:17:13.278009 18514 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.279582 18328 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:13.279420 18326 raft_consensus.cc:359] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.280086 18326 raft_consensus.cc:385] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.280158 18328 ts_tablet_manager.cc:1403] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.003s
I20260317 16:17:13.280779 18326 raft_consensus.cc:740] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.281383 18326 consensus_queue.cc:260] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.283315 18326 ts_tablet_manager.cc:1434] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260317 16:17:13.284276 18326 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:13.289242 18212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.289971 18326 tablet_bootstrap.cc:654] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.294422 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:13.296594 18103 consensus_peers.cc:597] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:13.300488 18103 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:13.313494 18328 raft_consensus.cc:359] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.314900 18328 raft_consensus.cc:385] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.315294 18328 raft_consensus.cc:740] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.316499 18328 consensus_queue.cc:260] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.322109 18504 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:13.313025 18559 raft_consensus.cc:2749] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.322942 18506 raft_consensus.cc:493] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.323734 18457 raft_consensus.cc:493] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.323652 18506 raft_consensus.cc:515] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.324191 18457 raft_consensus.cc:515] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.325711 18506 leader_election.cc:290] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:13.325991 18457 leader_election.cc:290] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:13.328260 18211 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.333029 18132 raft_consensus.cc:3060] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.333659 18328 ts_tablet_manager.cc:1434] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.053s user 0.003s sys 0.004s
I20260317 16:17:13.334702 18327 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:13.335278 18327 ts_tablet_manager.cc:1403] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.097s user 0.041s sys 0.044s
I20260317 16:17:13.338456 18327 raft_consensus.cc:359] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.339385 18327 raft_consensus.cc:385] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.339728 18327 raft_consensus.cc:740] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.340550 18327 consensus_queue.cc:260] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.343252 18327 ts_tablet_manager.cc:1434] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260317 16:17:13.344240 18327 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:13.334719 18328 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:13.350001 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.351560 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c94382dceadb48228dd3467bf673a11a" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.358064 18208 raft_consensus.cc:2468] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:13.361508 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:13.362182 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c94382dceadb48228dd3467bf673a11a" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:13.364494 18249 leader_election.cc:343] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:13.365556 18249 leader_election.cc:343] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.354579 18328 tablet_bootstrap.cc:654] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.372478 18417 consensus_queue.cc:1048] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:13.374207 18514 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 16:17:13.374955 18175 consensus_peers.cc:597] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:13.377053 18251 leader_election.cc:343] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.381547 18251 leader_election.cc:304] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.383016 18417 raft_consensus.cc:2749] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.383332 18251 leader_election.cc:304] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.390573 18327 tablet_bootstrap.cc:654] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.391885 18417 raft_consensus.cc:2804] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:13.394397 18417 raft_consensus.cc:493] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.395300 18417 raft_consensus.cc:3060] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.399472 18133 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.408985 18388 raft_consensus.cc:493] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.409837 18388 raft_consensus.cc:515] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.415215 18457 raft_consensus.cc:493] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.423521 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:13.420835 18457 raft_consensus.cc:515] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.444311 18417 raft_consensus.cc:515] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.450239 18504 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:13.453219 18326 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:13.453701 18326 ts_tablet_manager.cc:1403] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.170s user 0.007s sys 0.038s
I20260317 16:17:13.459877 18326 raft_consensus.cc:359] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.460804 18326 raft_consensus.cc:385] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.461125 18326 raft_consensus.cc:740] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.462059 18326 consensus_queue.cc:260] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.457267 18559 raft_consensus.cc:493] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.464594 18559 raft_consensus.cc:515] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.457268 18388 leader_election.cc:290] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:13.478650 18434 raft_consensus.cc:493] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.479411 18434 raft_consensus.cc:515] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.488191 18327 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:13.488293 18326 ts_tablet_manager.cc:1434] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
I20260317 16:17:13.488696 18327 ts_tablet_manager.cc:1403] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.145s user 0.019s sys 0.042s
I20260317 16:17:13.490226 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:13.491187 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:13.491844 18210 raft_consensus.cc:3060] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.491792 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.492573 18457 leader_election.cc:290] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:13.493021 18206 raft_consensus.cc:2468] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:13.496191 18251 leader_election.cc:304] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:13.497038 18457 raft_consensus.cc:2804] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:13.497408 18457 raft_consensus.cc:493] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.497736 18457 raft_consensus.cc:3060] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.503463 18205 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.508069 18205 raft_consensus.cc:2468] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:13.487900 18417 leader_election.cc:290] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:13.509977 18204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.511041 18327 raft_consensus.cc:359] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.526516 18327 raft_consensus.cc:385] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.526866 18327 raft_consensus.cc:740] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.527554 18327 consensus_queue.cc:260] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.512199 18102 leader_election.cc:304] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:13.513047 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:13.513983 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
W20260317 16:17:13.514626 18249 leader_election.cc:343] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.514745 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.523275 18434 leader_election.cc:290] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.508773 18559 leader_election.cc:290] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.524333 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:13.541361 18249 leader_election.cc:343] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.535086 18327 ts_tablet_manager.cc:1434] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.046s user 0.010s sys 0.001s
I20260317 16:17:13.534865 18514 raft_consensus.cc:2804] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 16:17:13.539319 18103 leader_election.cc:343] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.540854 18206 raft_consensus.cc:2468] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:13.550228 18514 raft_consensus.cc:493] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.550846 18514 raft_consensus.cc:3060] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.541484 18285 raft_consensus.cc:2468] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:13.529670 18326 tablet_bootstrap.cc:492] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:13.547253 18328 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:13.555557 18328 ts_tablet_manager.cc:1403] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.221s user 0.015s sys 0.034s
I20260317 16:17:13.556138 18251 leader_election.cc:304] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.563446 18328 raft_consensus.cc:359] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.564168 18328 raft_consensus.cc:385] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.564468 18328 raft_consensus.cc:740] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.566079 18328 consensus_queue.cc:260] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.567135 18327 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:13.567606 18102 leader_election.cc:304] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:13.559257 18417 raft_consensus.cc:2749] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.591185 18210 raft_consensus.cc:2468] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:13.580577 18417 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260317 16:17:13.594185 18559 raft_consensus.cc:2804] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:13.594624 18559 raft_consensus.cc:493] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.594985 18559 raft_consensus.cc:3060] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.602458 18251 leader_election.cc:304] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.603806 18417 raft_consensus.cc:2804] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.604417 18417 raft_consensus.cc:697] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:13.605472 18417 consensus_queue.cc:237] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:13.613183 18178 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 43845.9us
I20260317 16:17:13.612699 18555 raft_consensus.cc:493] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.625129 18328 ts_tablet_manager.cc:1434] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.069s user 0.015s sys 0.007s
I20260317 16:17:13.626212 18328 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
W20260317 16:17:13.616526 18102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 56352.7us
I20260317 16:17:13.633948 18434 raft_consensus.cc:493] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.634676 18434 raft_consensus.cc:515] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.628469 18555 raft_consensus.cc:515] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.613268 18514 raft_consensus.cc:515] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.640509 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c94382dceadb48228dd3467bf673a11a" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:13.660223 18434 leader_election.cc:290] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:13.663338 18175 leader_election.cc:343] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:13.673401 18103 consensus_peers.cc:597] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:13.676055 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:13.676342 18514 leader_election.cc:290] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.646170 18555 leader_election.cc:290] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.684406 18600 raft_consensus.cc:493] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.685736 18600 raft_consensus.cc:515] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.703939 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:13.704856 18282 raft_consensus.cc:3060] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.709962 18600 leader_election.cc:290] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.726724 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c94382dceadb48228dd3467bf673a11a" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:13.727941 18285 raft_consensus.cc:2468] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:13.728889 18178 leader_election.cc:304] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.666529 18457 raft_consensus.cc:515] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.729554 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:13.720785 18103 leader_election.cc:343] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.737186 18559 raft_consensus.cc:515] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.733772 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:13.745270 18178 leader_election.cc:343] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.737408 18600 raft_consensus.cc:2804] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:13.747351 18600 raft_consensus.cc:493] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.748342 18327 tablet_bootstrap.cc:654] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.747808 18600 raft_consensus.cc:3060] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.738405 18207 raft_consensus.cc:2468] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:13.753373 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:13.755546 18326 tablet_bootstrap.cc:654] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.754993 18206 raft_consensus.cc:3060] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.757647 18457 leader_election.cc:290] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:13.760097 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:13.760677 18211 raft_consensus.cc:3060] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.761508 18328 tablet_bootstrap.cc:654] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.764178 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
W20260317 16:17:13.766963 18249 leader_election.cc:343] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.768421 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:13.773612 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:13.774441 18103 leader_election.cc:343] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.773646 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:13.770460 18559 leader_election.cc:290] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.781663 18210 raft_consensus.cc:3060] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:13.784276 18175 leader_election.cc:343] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.786110 18282 raft_consensus.cc:2468] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:13.791122 18103 leader_election.cc:304] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:13.793457 18559 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:13.785499 18175 leader_election.cc:304] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.809090 18102 leader_election.cc:304] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:13.816888 18602 raft_consensus.cc:2804] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:13.817291 18602 raft_consensus.cc:493] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:13.825927 18602 raft_consensus.cc:3060] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.819494 18555 raft_consensus.cc:2749] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:13.817539 18559 raft_consensus.cc:2804] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.814200 18210 raft_consensus.cc:2468] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:13.813971 18211 raft_consensus.cc:2468] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:13.853582 18602 raft_consensus.cc:515] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.856392 18326 tablet_bootstrap.cc:492] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:13.856900 18326 ts_tablet_manager.cc:1403] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.354s user 0.012s sys 0.011s
I20260317 16:17:13.839092 18559 raft_consensus.cc:697] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:13.858085 18251 leader_election.cc:304] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.859436 18032 catalog_manager.cc:5671] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:13.860423 18502 raft_consensus.cc:2804] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.861413 18326 raft_consensus.cc:359] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.862129 18326 raft_consensus.cc:385] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.861860 18502 raft_consensus.cc:697] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:13.862869 18102 leader_election.cc:304] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:13.862473 18326 raft_consensus.cc:740] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.864893 18605 raft_consensus.cc:2804] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.867991 18326 consensus_queue.cc:260] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.870617 18326 ts_tablet_manager.cc:1434] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260317 16:17:13.870453 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:13.871079 18211 raft_consensus.cc:3060] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.873759 18600 raft_consensus.cc:515] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.875593 18602 leader_election.cc:290] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.865377 18605 raft_consensus.cc:697] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:13.875881 18326 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:13.864997 18502 consensus_queue.cc:237] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.877236 18328 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:13.876704 18605 consensus_queue.cc:237] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.877668 18328 ts_tablet_manager.cc:1403] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.252s user 0.021s sys 0.009s
I20260317 16:17:13.882987 18328 raft_consensus.cc:359] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.883484 18328 raft_consensus.cc:385] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.883692 18328 raft_consensus.cc:740] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.884174 18328 consensus_queue.cc:260] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.885973 18328 ts_tablet_manager.cc:1434] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260317 16:17:13.865902 18559 consensus_queue.cc:237] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.897047 18284 raft_consensus.cc:3060] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.891359 18206 raft_consensus.cc:2468] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:13.907073 18328 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:13.910392 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c94382dceadb48228dd3467bf673a11a" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:13.907217 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
W20260317 16:17:13.925614 18103 leader_election.cc:343] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:13.932384 18175 leader_election.cc:343] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:13.933156 18327 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:13.933657 18327 ts_tablet_manager.cc:1403] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.373s user 0.015s sys 0.003s
I20260317 16:17:13.939744 18211 raft_consensus.cc:2468] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:13.940022 18559 raft_consensus.cc:493] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:13.942117 18559 raft_consensus.cc:515] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.943050 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c94382dceadb48228dd3467bf673a11a" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:13.943621 18282 raft_consensus.cc:3060] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.945521 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:13.947228 18559 leader_election.cc:290] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.947849 18102 leader_election.cc:304] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:13.950466 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:13.951376 18211 raft_consensus.cc:2393] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:13.946525 18034 catalog_manager.cc:5671] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:13.952780 18102 leader_election.cc:304] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:13.953138 18327 raft_consensus.cc:359] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.955097 18327 raft_consensus.cc:385] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:13.937779 18600 leader_election.cc:290] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:13.955482 18327 raft_consensus.cc:740] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:13.957846 18327 consensus_queue.cc:260] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.953967 18602 raft_consensus.cc:2804] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.963302 18602 raft_consensus.cc:697] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:13.965407 18602 consensus_queue.cc:237] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:13.957574 18605 raft_consensus.cc:3060] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:13.971413 18326 tablet_bootstrap.cc:654] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:13.976082 18282 raft_consensus.cc:2468] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:13.980331 18178 leader_election.cc:304] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:13.983531 18555 raft_consensus.cc:2804] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:13.989140 18327 ts_tablet_manager.cc:1434] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.055s user 0.004s sys 0.005s
I20260317 16:17:13.985193 18555 raft_consensus.cc:697] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:13.993964 18555 consensus_queue.cc:237] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.007846 18327 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.020056 18502 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.020992 18502 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.025792 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:14.027813 18249 leader_election.cc:343] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.033941 18502 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:14.035158 18555 raft_consensus.cc:493] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.035667 18555 raft_consensus.cc:515] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.038589 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.039319 18211 raft_consensus.cc:2468] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:14.041035 18251 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.042189 18504 raft_consensus.cc:2804] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:14.042390 18328 tablet_bootstrap.cc:654] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.042595 18504 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:14.040108 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.043005 18504 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.045745 18555 leader_election.cc:290] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.053459 18326 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.053995 18326 ts_tablet_manager.cc:1403] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.178s user 0.014s sys 0.004s
I20260317 16:17:14.057067 18504 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.057288 18326 raft_consensus.cc:359] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.058478 18326 raft_consensus.cc:385] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.058821 18326 raft_consensus.cc:740] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.059501 18326 consensus_queue.cc:260] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.060576 18605 raft_consensus.cc:2749] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.061838 18326 ts_tablet_manager.cc:1434] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260317 16:17:14.064881 18175 leader_election.cc:343] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.046268 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.068023 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
W20260317 16:17:14.068461 18178 leader_election.cc:343] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.069041 18178 leader_election.cc:304] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
W20260317 16:17:14.069293 18249 leader_election.cc:343] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:14.066877 18613 log.cc:927] Time spent T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.072s user 0.004s sys 0.000s
I20260317 16:17:14.070031 18033 catalog_manager.cc:5671] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:14.075624 18326 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.077615 18601 raft_consensus.cc:2749] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.079074 18504 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:14.093039 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:14.093606 18326 tablet_bootstrap.cc:654] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.093675 18211 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.098904 18327 tablet_bootstrap.cc:654] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.103400 18328 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:14.104004 18328 ts_tablet_manager.cc:1403] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.197s user 0.012s sys 0.018s
I20260317 16:17:14.101104 18504 consensus_queue.cc:1048] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260317 16:17:14.110040 18211 raft_consensus.cc:2468] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:14.111658 18251 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.112771 18502 raft_consensus.cc:2804] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:14.113262 18502 raft_consensus.cc:697] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:14.114105 18502 consensus_queue.cc:237] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.122992 18504 raft_consensus.cc:493] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.123720 18504 raft_consensus.cc:515] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.126302 18504 leader_election.cc:290] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:14.131071 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.132249 18328 raft_consensus.cc:359] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:14.132853 18249 leader_election.cc:343] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.133193 18328 raft_consensus.cc:385] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.133545 18328 raft_consensus.cc:740] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.134888 18328 consensus_queue.cc:260] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.149540 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:14.152169 18251 leader_election.cc:343] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.154124 18251 leader_election.cc:304] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.156095 18504 raft_consensus.cc:2749] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.163982 18327 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.164855 18327 ts_tablet_manager.cc:1403] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.157s user 0.018s sys 0.007s
I20260317 16:17:14.166466 18032 catalog_manager.cc:5671] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:14.169211 18032 catalog_manager.cc:5671] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:14.176556 18328 ts_tablet_manager.cc:1434] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.072s user 0.008s sys 0.004s
I20260317 16:17:14.178470 18326 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.179019 18326 ts_tablet_manager.cc:1403] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.104s user 0.042s sys 0.013s
I20260317 16:17:14.178642 18034 catalog_manager.cc:5671] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:14.188900 18504 consensus_queue.cc:1048] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.189702 18326 raft_consensus.cc:359] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.190443 18326 raft_consensus.cc:385] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.190719 18326 raft_consensus.cc:740] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.190857 18328 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:14.191421 18326 consensus_queue.cc:260] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.196197 18326 ts_tablet_manager.cc:1434] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260317 16:17:14.197085 18326 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.187844 18327 raft_consensus.cc:359] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.199236 18327 raft_consensus.cc:385] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.199533 18327 raft_consensus.cc:740] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.200230 18327 consensus_queue.cc:260] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.202458 18327 ts_tablet_manager.cc:1434] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.037s user 0.003s sys 0.002s
I20260317 16:17:14.203365 18327 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.204946 18326 tablet_bootstrap.cc:654] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.216307 18326 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.216933 18326 ts_tablet_manager.cc:1403] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260317 16:17:14.222570 18326 raft_consensus.cc:359] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.223346 18326 raft_consensus.cc:385] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.223639 18326 raft_consensus.cc:740] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.224366 18326 consensus_queue.cc:260] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.226547 18326 ts_tablet_manager.cc:1434] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260317 16:17:14.227526 18326 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.233117 18326 tablet_bootstrap.cc:654] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.235474 18328 tablet_bootstrap.cc:654] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.249691 18327 tablet_bootstrap.cc:654] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.250465 18326 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.251055 18326 ts_tablet_manager.cc:1403] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.013s
I20260317 16:17:14.254678 18326 raft_consensus.cc:359] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.255426 18326 raft_consensus.cc:385] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.255705 18326 raft_consensus.cc:740] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.256512 18326 consensus_queue.cc:260] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.258376 18328 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:14.258975 18328 ts_tablet_manager.cc:1403] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.001s
I20260317 16:17:14.259181 18326 ts_tablet_manager.cc:1434] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260317 16:17:14.260149 18326 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.261760 18328 raft_consensus.cc:359] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.262436 18328 raft_consensus.cc:385] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.262712 18328 raft_consensus.cc:740] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.263537 18328 consensus_queue.cc:260] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.265170 18327 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.265631 18327 ts_tablet_manager.cc:1403] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.002s
I20260317 16:17:14.266168 18328 ts_tablet_manager.cc:1434] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:17:14.266219 18326 tablet_bootstrap.cc:654] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.267280 18328 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:14.268585 18327 raft_consensus.cc:359] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.269239 18327 raft_consensus.cc:385] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.269474 18327 raft_consensus.cc:740] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.274747 18328 tablet_bootstrap.cc:654] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.278645 18326 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.279171 18326 ts_tablet_manager.cc:1403] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.003s
I20260317 16:17:14.282341 18326 raft_consensus.cc:359] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.283581 18326 raft_consensus.cc:385] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.283257 18327 consensus_queue.cc:260] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.284039 18326 raft_consensus.cc:740] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.284809 18326 consensus_queue.cc:260] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.287138 18327 ts_tablet_manager.cc:1434] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260317 16:17:14.287856 18326 ts_tablet_manager.cc:1434] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260317 16:17:14.288153 18327 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.288810 18326 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
W20260317 16:17:14.290023 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:14.301570 18326 tablet_bootstrap.cc:654] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.306376 18032 catalog_manager.cc:5671] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:14.308303 18327 tablet_bootstrap.cc:654] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.315167 18327 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.315689 18327 ts_tablet_manager.cc:1403] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.008s
I20260317 16:17:14.315865 18328 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:14.316367 18328 ts_tablet_manager.cc:1403] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.049s user 0.019s sys 0.020s
I20260317 16:17:14.318348 18327 raft_consensus.cc:359] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.319298 18327 raft_consensus.cc:385] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.319591 18327 raft_consensus.cc:740] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.319376 18328 raft_consensus.cc:359] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.320111 18328 raft_consensus.cc:385] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.320360 18328 raft_consensus.cc:740] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.320319 18327 consensus_queue.cc:260] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.321359 18328 consensus_queue.cc:260] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.321791 18559 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.322266 18559 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.324362 18559 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.325134 18327 ts_tablet_manager.cc:1434] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260317 16:17:14.326082 18327 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.326274 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.326946 18211 raft_consensus.cc:2393] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 021d78130690452ba094f4a392fb4378 in this term.
I20260317 16:17:14.330120 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.331049 18559 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.332885 18326 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.333325 18326 ts_tablet_manager.cc:1403] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.003s
I20260317 16:17:14.335615 18103 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:14.336462 18326 raft_consensus.cc:359] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.337190 18326 raft_consensus.cc:385] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.337535 18326 raft_consensus.cc:740] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.334420 18502 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260317 16:17:14.338248 18326 consensus_queue.cc:260] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.339636 18327 tablet_bootstrap.cc:654] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.343374 18602 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.351974 18327 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.352612 18326 ts_tablet_manager.cc:1434] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260317 16:17:14.353705 18327 ts_tablet_manager.cc:1403] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.000s
I20260317 16:17:14.354300 18326 tablet_bootstrap.cc:492] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.356233 18133 raft_consensus.cc:3060] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.356804 18327 raft_consensus.cc:359] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.357517 18327 raft_consensus.cc:385] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.357793 18327 raft_consensus.cc:740] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.359419 18327 consensus_queue.cc:260] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.362345 18326 tablet_bootstrap.cc:654] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:14.363307 18103 consensus_peers.cc:597] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:14.363987 18327 ts_tablet_manager.cc:1434] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260317 16:17:14.364976 18327 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.371135 18605 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.373411 18326 tablet_bootstrap.cc:492] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.373878 18326 ts_tablet_manager.cc:1403] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260317 16:17:14.374956 18327 tablet_bootstrap.cc:654] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.376753 18326 raft_consensus.cc:359] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.377445 18326 raft_consensus.cc:385] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.377697 18326 raft_consensus.cc:740] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.377657 18559 consensus_queue.cc:1048] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:14.379236 18326 consensus_queue.cc:260] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.384244 18326 ts_tablet_manager.cc:1434] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260317 16:17:14.385259 18326 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.392859 18326 tablet_bootstrap.cc:654] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.393651 18602 raft_consensus.cc:2749] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.396832 18327 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.397328 18327 ts_tablet_manager.cc:1403] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.011s
I20260317 16:17:14.400934 18327 raft_consensus.cc:359] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.401628 18327 raft_consensus.cc:385] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.401930 18327 raft_consensus.cc:740] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.401630 18328 ts_tablet_manager.cc:1434] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s
I20260317 16:17:14.402511 18327 consensus_queue.cc:260] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.403618 18328 tablet_bootstrap.cc:492] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:14.406142 18434 raft_consensus.cc:493] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.407140 18434 raft_consensus.cc:515] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.409147 18434 leader_election.cc:290] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.410591 18328 tablet_bootstrap.cc:654] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.411118 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.423363 18328 tablet_bootstrap.cc:492] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:14.423775 18327 ts_tablet_manager.cc:1434] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.026s user 0.010s sys 0.000s
I20260317 16:17:14.423379 18137 raft_consensus.cc:3060] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.424841 18328 ts_tablet_manager.cc:1403] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20260317 16:17:14.425563 18327 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.429795 18328 raft_consensus.cc:359] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.430580 18328 raft_consensus.cc:385] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.430904 18328 raft_consensus.cc:740] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.431789 18328 consensus_queue.cc:260] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.433977 18328 ts_tablet_manager.cc:1434] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260317 16:17:14.437243 18327 tablet_bootstrap.cc:654] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.440574 18328 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:14.443101 18326 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.443545 18326 ts_tablet_manager.cc:1403] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.004s
I20260317 16:17:14.444829 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.446018 18326 raft_consensus.cc:359] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.446650 18326 raft_consensus.cc:385] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.446632 18417 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:14.447176 18326 raft_consensus.cc:740] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.448129 18326 consensus_queue.cc:260] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.450555 18326 ts_tablet_manager.cc:1434] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260317 16:17:14.450593 18103 consensus_peers.cc:597] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:14.452171 18326 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:14.452951 18103 leader_election.cc:304] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:14.454218 18602 raft_consensus.cc:2749] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.454443 18600 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.456169 18328 tablet_bootstrap.cc:654] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.458964 18326 tablet_bootstrap.cc:654] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.486577 18327 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.487164 18327 ts_tablet_manager.cc:1403] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.009s
I20260317 16:17:14.489331 18600 raft_consensus.cc:493] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.489857 18600 raft_consensus.cc:515] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.491343 18555 raft_consensus.cc:493] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.492515 18600 leader_election.cc:290] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.491952 18555 raft_consensus.cc:515] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.502425 18327 raft_consensus.cc:359] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.503170 18327 raft_consensus.cc:385] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.503485 18327 raft_consensus.cc:740] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.503207 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.504096 18602 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.504985 18327 consensus_queue.cc:260] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.498270 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.507405 18327 ts_tablet_manager.cc:1434] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260317 16:17:14.509959 18327 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.506610 18132 raft_consensus.cc:2468] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:14.511427 18175 leader_election.cc:304] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:14.512020 18555 leader_election.cc:290] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.512593 18555 raft_consensus.cc:2804] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:14.512964 18555 raft_consensus.cc:493] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:14.513324 18555 raft_consensus.cc:3060] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.515062 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.516016 18132 raft_consensus.cc:2468] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:14.516927 18327 tablet_bootstrap.cc:654] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.504139 18284 raft_consensus.cc:2468] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:14.518524 18175 leader_election.cc:304] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:14.516927 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.519769 18601 raft_consensus.cc:2804] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:14.520258 18601 raft_consensus.cc:493] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:14.520598 18601 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.522231 18417 raft_consensus.cc:493] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:17:14.522405 18178 leader_election.cc:343] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.523037 18417 raft_consensus.cc:515] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.525398 18417 leader_election.cc:290] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:14.526654 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.530117 18327 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.530601 18327 ts_tablet_manager.cc:1403] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.003s
I20260317 16:17:14.531014 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75ecbd9bfe9e4fe4b1f5146d5029d1c0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.531283 18417 raft_consensus.cc:493] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.531879 18417 raft_consensus.cc:515] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.533453 18251 leader_election.cc:304] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.533708 18327 raft_consensus.cc:359] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.534463 18327 raft_consensus.cc:385] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.534595 18504 raft_consensus.cc:2749] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.534799 18327 raft_consensus.cc:740] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.535380 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.536300 18417 leader_election.cc:290] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:14.536914 18249 leader_election.cc:343] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.537392 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.543251 18206 raft_consensus.cc:2468] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:14.543697 18135 raft_consensus.cc:3060] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.535625 18327 consensus_queue.cc:260] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.544588 18251 leader_election.cc:304] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.545563 18417 raft_consensus.cc:2804] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:14.546084 18417 raft_consensus.cc:493] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:14.546487 18417 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.551592 18555 raft_consensus.cc:515] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.553264 18327 ts_tablet_manager.cc:1434] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260317 16:17:14.553670 18555 leader_election.cc:290] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.554728 18327 tablet_bootstrap.cc:492] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.555087 18605 raft_consensus.cc:493] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.555584 18605 raft_consensus.cc:515] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.541177 18601 raft_consensus.cc:515] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.557668 18605 leader_election.cc:290] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.558712 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.559396 18601 leader_election.cc:290] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.559588 18284 raft_consensus.cc:2468] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:14.561436 18103 leader_election.cc:304] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:14.563603 18417 raft_consensus.cc:515] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.566468 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:14.579589 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:14.579918 18132 raft_consensus.cc:3060] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.563659 18457 raft_consensus.cc:493] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:17:14.581602 18178 leader_election.cc:343] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.568357 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:14.583505 18134 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.583837 18327 tablet_bootstrap.cc:654] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.568440 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.586704 18457 raft_consensus.cc:515] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.569391 18605 raft_consensus.cc:2804] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:14.570364 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:14.590497 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:14.590652 18605 raft_consensus.cc:493] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:14.590837 18284 raft_consensus.cc:3060] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.591099 18605 raft_consensus.cc:3060] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.602144 18457 leader_election.cc:290] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:14.607529 18249 leader_election.cc:343] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.611001 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.612149 18600 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.613171 18328 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:14.613605 18328 ts_tablet_manager.cc:1403] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.173s user 0.014s sys 0.005s
I20260317 16:17:14.614168 18132 raft_consensus.cc:2468] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:14.614591 18210 raft_consensus.cc:2468] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:14.642729 18175 leader_election.cc:304] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:14.616451 18134 raft_consensus.cc:2468] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:14.619112 18417 leader_election.cc:290] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:14.625471 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:14.645881 18207 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.645881 18555 raft_consensus.cc:2804] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:14.646895 18555 raft_consensus.cc:697] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:14.647696 18555 consensus_queue.cc:237] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.647059 18175 leader_election.cc:304] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:14.651846 18600 raft_consensus.cc:2804] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:14.652352 18600 raft_consensus.cc:697] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:14.653074 18207 raft_consensus.cc:2468] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:14.653139 18600 consensus_queue.cc:237] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.627847 18328 raft_consensus.cc:359] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.655238 18328 raft_consensus.cc:385] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.655620 18328 raft_consensus.cc:740] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.628755 18502 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:14.635180 18654 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.611052 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.637275 18605 raft_consensus.cc:515] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.639146 18327 tablet_bootstrap.cc:492] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.639907 18285 raft_consensus.cc:3060] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.661674 18327 ts_tablet_manager.cc:1403] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.016s
I20260317 16:17:14.640091 18284 raft_consensus.cc:2468] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:14.641650 18326 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.662397 18326 ts_tablet_manager.cc:1403] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.210s user 0.029s sys 0.044s
I20260317 16:17:14.664260 18328 consensus_queue.cc:260] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.662739 18559 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.659744 18654 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.683008 18328 ts_tablet_manager.cc:1434] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20260317 16:17:14.684567 18249 leader_election.cc:304] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.691267 18327 raft_consensus.cc:359] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.691960 18327 raft_consensus.cc:385] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.692257 18327 raft_consensus.cc:740] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.695384 18328 tablet_bootstrap.cc:492] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:14.697000 18251 leader_election.cc:304] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.696045 18327 consensus_queue.cc:260] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.703177 18326 raft_consensus.cc:359] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.704203 18326 raft_consensus.cc:385] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.704542 18326 raft_consensus.cc:740] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.690114 18504 raft_consensus.cc:493] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.691139 18506 raft_consensus.cc:2749] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.705282 18326 consensus_queue.cc:260] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.681107 18559 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.721951 18559 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.724193 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:14.725612 18103 leader_election.cc:343] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.717684 18506 raft_consensus.cc:2804] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:14.728412 18506 raft_consensus.cc:697] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:14.729251 18506 consensus_queue.cc:237] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.731611 18654 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.710918 18504 raft_consensus.cc:515] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.733417 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.734099 18134 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:14.736295 18175 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:14.738595 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.743099 18504 leader_election.cc:290] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:14.747347 18178 leader_election.cc:343] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.751556 18326 ts_tablet_manager.cc:1434] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.089s user 0.008s sys 0.000s
I20260317 16:17:14.758719 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b451ac456d424629ae5ce6040cb15bd2" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.758947 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b451ac456d424629ae5ce6040cb15bd2" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.750613 18417 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260317 16:17:14.760239 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:14.760879 18284 raft_consensus.cc:3060] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.749580 18663 raft_consensus.cc:2804] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:14.755630 18605 leader_election.cc:290] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.762049 18663 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:14.762431 18663 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.765420 18326 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
W20260317 16:17:14.766042 18249 leader_election.cc:343] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.769315 18284 raft_consensus.cc:2468] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:14.770606 18103 leader_election.cc:304] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
W20260317 16:17:14.770849 18251 leader_election.cc:343] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.771261 18251 leader_election.cc:304] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.772686 18559 raft_consensus.cc:2804] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:14.773988 18559 raft_consensus.cc:697] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:14.775595 18559 consensus_queue.cc:237] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.772197 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:14.776475 18210 raft_consensus.cc:3060] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.779040 18502 raft_consensus.cc:2749] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.787292 18327 ts_tablet_manager.cc:1434] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.125s user 0.012s sys 0.009s
I20260317 16:17:14.782333 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.794598 18664 raft_consensus.cc:493] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.795243 18664 raft_consensus.cc:515] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.797462 18327 tablet_bootstrap.cc:492] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.803437 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.807024 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:14.809230 18249 leader_election.cc:343] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.801994 18664 leader_election.cc:290] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:14.813040 18251 leader_election.cc:343] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.811494 18210 raft_consensus.cc:2468] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:14.813934 18251 leader_election.cc:304] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.815752 18032 catalog_manager.cc:5671] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:14.818413 18032 catalog_manager.cc:5671] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:14.811846 18328 tablet_bootstrap.cc:654] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.825331 18555 raft_consensus.cc:493] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.836860 18665 raft_consensus.cc:2749] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.839325 18285 raft_consensus.cc:3060] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.835048 18555 raft_consensus.cc:515] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.841480 18555 leader_election.cc:290] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:14.843684 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:14.845470 18175 leader_election.cc:343] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.848070 18663 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.849252 18206 raft_consensus.cc:2393] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:14.852370 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:14.852859 18102 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:14.853412 18663 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:14.853469 18178 leader_election.cc:343] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.854084 18178 leader_election.cc:304] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.856844 18603 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:14.858953 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
W20260317 16:17:14.861934 18178 leader_election.cc:343] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.860306 18663 raft_consensus.cc:2749] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.873757 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:14.878306 18603 raft_consensus.cc:2749] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.889123 18135 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:14.898823 18666 raft_consensus.cc:493] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:17:14.900760 18667 log.cc:927] Time spent T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260317 16:17:14.900663 18666 raft_consensus.cc:515] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.902801 18666 leader_election.cc:290] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:14.903394 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c9d82ef74c442299f4a3cf04a5bb2d7" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:14.904788 18249 leader_election.cc:343] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.907271 18326 tablet_bootstrap.cc:654] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.908778 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c9d82ef74c442299f4a3cf04a5bb2d7" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.909188 18175 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
W20260317 16:17:14.910449 18251 leader_election.cc:343] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.910552 18555 raft_consensus.cc:2804] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:14.910909 18251 leader_election.cc:304] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.911077 18555 raft_consensus.cc:697] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:14.911890 18670 raft_consensus.cc:2749] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.913352 18555 consensus_queue.cc:237] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.914094 18327 tablet_bootstrap.cc:654] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.920370 18670 raft_consensus.cc:493] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:14.920921 18670 raft_consensus.cc:515] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.924978 18670 leader_election.cc:290] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:14.927181 18672 log.cc:927] Time spent T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260317 16:17:14.928215 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2a036705e4a4da5a5392d119edb244e" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:14.929327 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2a036705e4a4da5a5392d119edb244e" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:14.923161 18328 tablet_bootstrap.cc:492] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:14.934283 18328 ts_tablet_manager.cc:1403] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.243s user 0.016s sys 0.003s
W20260317 16:17:14.941609 18249 leader_election.cc:343] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:14.942574 18251 leader_election.cc:343] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:14.942974 18251 leader_election.cc:304] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:14.941999 18328 raft_consensus.cc:359] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.947333 18328 raft_consensus.cc:385] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.947743 18328 raft_consensus.cc:740] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.948963 18328 consensus_queue.cc:260] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.948393 18666 raft_consensus.cc:2749] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:14.963539 18328 ts_tablet_manager.cc:1434] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20260317 16:17:14.964658 18328 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:14.965344 18326 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:14.965791 18326 ts_tablet_manager.cc:1403] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.013s
I20260317 16:17:14.973454 18035 catalog_manager.cc:5671] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:14.985131 18327 tablet_bootstrap.cc:492] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:14.985636 18327 ts_tablet_manager.cc:1403] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.188s user 0.010s sys 0.022s
I20260317 16:17:14.988483 18327 raft_consensus.cc:359] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.989153 18327 raft_consensus.cc:385] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:14.989476 18327 raft_consensus.cc:740] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:14.989655 18328 tablet_bootstrap.cc:654] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:14.990139 18327 consensus_queue.cc:260] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.993095 18327 ts_tablet_manager.cc:1434] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260317 16:17:14.995149 18327 tablet_bootstrap.cc:492] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:14.999997 18555 raft_consensus.cc:493] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.000535 18555 raft_consensus.cc:515] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:14.997918 18031 catalog_manager.cc:5671] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.002609 18555 leader_election.cc:290] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.005152 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77a63cdb5604ab493b7f5f9dc921763" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.006271 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77a63cdb5604ab493b7f5f9dc921763" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.007035 18135 raft_consensus.cc:2468] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:15.008634 18327 tablet_bootstrap.cc:654] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:15.009730 18178 leader_election.cc:343] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.010797 18175 leader_election.cc:304] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:15.011858 18663 raft_consensus.cc:2804] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.012248 18663 raft_consensus.cc:493] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.012821 18663 raft_consensus.cc:3060] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.017217 18328 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:15.018054 18328 ts_tablet_manager.cc:1403] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.017s
I20260317 16:17:15.022869 18328 raft_consensus.cc:359] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.023545 18328 raft_consensus.cc:385] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.023792 18328 raft_consensus.cc:740] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.024482 18328 consensus_queue.cc:260] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.026980 18328 ts_tablet_manager.cc:1434] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.009s user 0.007s sys 0.004s
I20260317 16:17:15.027911 18328 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:15.035046 18328 tablet_bootstrap.cc:654] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.038987 18032 catalog_manager.cc:5671] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.057796 18326 raft_consensus.cc:359] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.058583 18326 raft_consensus.cc:385] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.062489 18326 raft_consensus.cc:740] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.063563 18326 consensus_queue.cc:260] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.065992 18326 ts_tablet_manager.cc:1434] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.100s user 0.003s sys 0.003s
I20260317 16:17:15.071062 18663 raft_consensus.cc:515] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.074023 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77a63cdb5604ab493b7f5f9dc921763" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
W20260317 16:17:15.075161 18178 leader_election.cc:343] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.075346 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77a63cdb5604ab493b7f5f9dc921763" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:15.075922 18135 raft_consensus.cc:3060] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.076140 18326 tablet_bootstrap.cc:492] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:15.082813 18663 leader_election.cc:290] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.087312 18327 tablet_bootstrap.cc:492] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.087975 18327 ts_tablet_manager.cc:1403] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.093s user 0.040s sys 0.039s
I20260317 16:17:15.101045 18135 raft_consensus.cc:2468] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:15.105752 18175 leader_election.cc:304] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:15.110976 18556 raft_consensus.cc:2804] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:15.111507 18556 raft_consensus.cc:697] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:15.112502 18556 consensus_queue.cc:237] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.136148 18328 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:15.136590 18328 ts_tablet_manager.cc:1403] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.109s user 0.016s sys 0.018s
I20260317 16:17:15.136571 18327 raft_consensus.cc:359] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.137365 18327 raft_consensus.cc:385] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.137629 18327 raft_consensus.cc:740] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.138229 18327 consensus_queue.cc:260] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.142228 18326 tablet_bootstrap.cc:654] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.143337 18328 raft_consensus.cc:359] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.144196 18328 raft_consensus.cc:385] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.144518 18328 raft_consensus.cc:740] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.145309 18328 consensus_queue.cc:260] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.146589 18327 ts_tablet_manager.cc:1434] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.058s user 0.013s sys 0.035s
I20260317 16:17:15.147516 18327 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:15.147548 18328 ts_tablet_manager.cc:1434] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260317 16:17:15.150274 18328 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:15.152472 18326 tablet_bootstrap.cc:492] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:15.152957 18326 ts_tablet_manager.cc:1403] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.001s
I20260317 16:17:15.158004 18327 tablet_bootstrap.cc:654] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:15.154572 18178 consensus_peers.cc:597] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:15.158708 18328 tablet_bootstrap.cc:654] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.159044 18326 raft_consensus.cc:359] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.159754 18326 raft_consensus.cc:385] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.160012 18326 raft_consensus.cc:740] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.160635 18326 consensus_queue.cc:260] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.166007 18326 ts_tablet_manager.cc:1434] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260317 16:17:15.166961 18600 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.167471 18600 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.170271 18327 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.169859 18600 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.170898 18327 ts_tablet_manager.cc:1403] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.010s
I20260317 16:17:15.171438 18326 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:15.171389 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:15.172719 18178 leader_election.cc:343] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.174183 18327 raft_consensus.cc:359] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.176673 18327 raft_consensus.cc:385] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.176945 18327 raft_consensus.cc:740] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.177536 18514 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.177824 18326 tablet_bootstrap.cc:654] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.177654 18327 consensus_queue.cc:260] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.178103 18328 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:15.178699 18328 ts_tablet_manager.cc:1403] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.007s
I20260317 16:17:15.178062 18514 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.180291 18327 ts_tablet_manager.cc:1434] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260317 16:17:15.181294 18327 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:15.181233 18514 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.181505 18328 raft_consensus.cc:359] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.182322 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.182875 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.183328 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.183437 18207 raft_consensus.cc:2468] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:15.183931 18135 raft_consensus.cc:2468] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:15.184903 18175 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:15.185461 18102 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:15.185920 18600 raft_consensus.cc:2804] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.186287 18600 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.186451 18514 raft_consensus.cc:2804] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.186614 18600 raft_consensus.cc:3060] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.187083 18514 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.187688 18514 raft_consensus.cc:3060] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.189518 18327 tablet_bootstrap.cc:654] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
W20260317 16:17:15.191509 18103 leader_election.cc:343] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.191947 18328 raft_consensus.cc:385] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.192348 18328 raft_consensus.cc:740] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.193073 18328 consensus_queue.cc:260] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.200855 18663 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:15.203747 18600 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.203174 18031 catalog_manager.cc:5671] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.206297 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:15.206748 18600 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.207926 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:15.212137 18328 ts_tablet_manager.cc:1434] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20260317 16:17:15.218868 18328 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
W20260317 16:17:15.222241 18178 leader_election.cc:343] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.215335 18514 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.224658 18665 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:15.225173 18137 raft_consensus.cc:2393] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b3a81306894b71bf2eacc88c770eaa in current term 1: Already voted for candidate 94edf207909c4f8990bfe29ed1a87a6d in this term.
I20260317 16:17:15.229612 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:15.230472 18210 raft_consensus.cc:2393] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:15.231501 18175 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.232419 18600 raft_consensus.cc:2749] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.234974 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:15.237334 18137 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:17:15.244560 18103 leader_election.cc:343] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.245380 18103 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:15.247063 18605 raft_consensus.cc:2749] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.253191 18660 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:15.254506 18514 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.259869 18660 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:15.268826 18665 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:15.274083 18328 tablet_bootstrap.cc:654] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.278419 18327 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.280705 18327 ts_tablet_manager.cc:1403] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.100s user 0.026s sys 0.046s
I20260317 16:17:15.284356 18327 raft_consensus.cc:359] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.285032 18327 raft_consensus.cc:385] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.285290 18327 raft_consensus.cc:740] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.307086 18600 raft_consensus.cc:493] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.307646 18600 raft_consensus.cc:515] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.309624 18600 leader_election.cc:290] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.313452 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b451ac456d424629ae5ce6040cb15bd2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.312285 18603 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260317 16:17:15.315016 18175 leader_election.cc:343] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.316008 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b451ac456d424629ae5ce6040cb15bd2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.316721 18285 raft_consensus.cc:2468] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:15.318114 18178 leader_election.cc:304] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.320091 18556 raft_consensus.cc:2804] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.320221 18666 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.320498 18556 raft_consensus.cc:493] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.320850 18556 raft_consensus.cc:3060] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.320693 18666 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.323294 18666 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.324433 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.325560 18137 raft_consensus.cc:2393] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:15.325889 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.328167 18327 consensus_queue.cc:260] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.330175 18249 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.331509 18666 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.336941 18556 raft_consensus.cc:515] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.339658 18327 ts_tablet_manager.cc:1434] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.059s user 0.004s sys 0.003s
I20260317 16:17:15.339901 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b451ac456d424629ae5ce6040cb15bd2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:15.341250 18327 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
W20260317 16:17:15.341989 18175 leader_election.cc:343] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.344930 18666 raft_consensus.cc:2749] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.345239 18514 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:15.349243 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b451ac456d424629ae5ce6040cb15bd2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:15.349462 18556 leader_election.cc:290] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.349845 18285 raft_consensus.cc:3060] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.353219 18665 raft_consensus.cc:493] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.353920 18665 raft_consensus.cc:515] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.361183 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.364637 18328 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:15.365149 18328 ts_tablet_manager.cc:1403] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.147s user 0.016s sys 0.006s
I20260317 16:17:15.368661 18285 raft_consensus.cc:2468] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:15.370563 18178 leader_election.cc:304] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.371217 18328 raft_consensus.cc:359] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.371661 18555 raft_consensus.cc:2804] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:15.371899 18328 raft_consensus.cc:385] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.372179 18328 raft_consensus.cc:740] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.372148 18555 raft_consensus.cc:697] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:15.373142 18555 consensus_queue.cc:237] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.381642 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44eefa978130466ead1b5828a0e0c4f3" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.382066 18327 tablet_bootstrap.cc:654] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.383477 18285 raft_consensus.cc:3060] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.392172 18328 consensus_queue.cc:260] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.394620 18328 ts_tablet_manager.cc:1434] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260317 16:17:15.395690 18251 leader_election.cc:304] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.397328 18600 raft_consensus.cc:493] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.397883 18600 raft_consensus.cc:515] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.400132 18600 leader_election.cc:290] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.401058 18417 raft_consensus.cc:493] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.402201 18417 raft_consensus.cc:515] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.404026 18417 leader_election.cc:290] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.406394 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.408812 18326 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:15.408553 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77a63cdb5604ab493b7f5f9dc921763" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.409566 18136 raft_consensus.cc:2393] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:15.409339 18326 ts_tablet_manager.cc:1403] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.238s user 0.034s sys 0.075s
W20260317 16:17:15.411042 18175 leader_election.cc:343] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.420768 18665 leader_election.cc:290] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.421329 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.422063 18282 raft_consensus.cc:2468] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:15.423164 18178 leader_election.cc:304] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.425170 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77a63cdb5604ab493b7f5f9dc921763" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.437235 18555 raft_consensus.cc:2804] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.432178 18327 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.440498 18327 ts_tablet_manager.cc:1403] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.008s
I20260317 16:17:15.395691 18328 tablet_bootstrap.cc:492] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:15.440698 18555 raft_consensus.cc:493] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.441507 18555 raft_consensus.cc:3060] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.445304 18327 raft_consensus.cc:359] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.426595 18326 raft_consensus.cc:359] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.446723 18327 raft_consensus.cc:385] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.446979 18326 raft_consensus.cc:385] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.447139 18327 raft_consensus.cc:740] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.447345 18326 raft_consensus.cc:740] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.447872 18327 consensus_queue.cc:260] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.448068 18326 consensus_queue.cc:260] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.450246 18327 ts_tablet_manager.cc:1434] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20260317 16:17:15.450827 18326 ts_tablet_manager.cc:1434] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.040s user 0.004s sys 0.003s
I20260317 16:17:15.451259 18327 tablet_bootstrap.cc:492] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:15.451923 18326 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:15.452330 18251 leader_election.cc:304] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.456459 18328 tablet_bootstrap.cc:654] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.456357 18502 raft_consensus.cc:2749] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.461675 18665 raft_consensus.cc:3060] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.461825 18326 tablet_bootstrap.cc:654] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.459810 18327 tablet_bootstrap.cc:654] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.471701 18556 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:15.473198 18327 tablet_bootstrap.cc:492] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.470468 18665 raft_consensus.cc:2749] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.473686 18327 ts_tablet_manager.cc:1403] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.010s
I20260317 16:17:15.481396 18328 tablet_bootstrap.cc:492] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:15.481930 18328 ts_tablet_manager.cc:1403] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.006s
I20260317 16:17:15.485026 18035 catalog_manager.cc:5671] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.487605 18327 raft_consensus.cc:359] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.488307 18327 raft_consensus.cc:385] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.488574 18327 raft_consensus.cc:740] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.489171 18327 consensus_queue.cc:260] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.491397 18327 ts_tablet_manager.cc:1434] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260317 16:17:15.492339 18327 tablet_bootstrap.cc:492] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:15.484606 18328 raft_consensus.cc:359] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.505752 18328 raft_consensus.cc:385] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.506266 18328 raft_consensus.cc:740] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.507134 18328 consensus_queue.cc:260] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.498651 18514 raft_consensus.cc:493] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.508984 18514 raft_consensus.cc:515] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.510499 18514 leader_election.cc:290] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.511932 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c9d82ef74c442299f4a3cf04a5bb2d7" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.512534 18285 raft_consensus.cc:2468] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:15.513664 18103 leader_election.cc:304] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:15.522087 18665 raft_consensus.cc:493] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.523663 18665 raft_consensus.cc:515] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.526420 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c9d82ef74c442299f4a3cf04a5bb2d7" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.527135 18665 leader_election.cc:290] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.516455 18603 raft_consensus.cc:2804] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.529178 18603 raft_consensus.cc:493] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.527276 18210 raft_consensus.cc:2468] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:15.529554 18603 raft_consensus.cc:3060] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.516670 18600 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260317 16:17:15.459093 18555 raft_consensus.cc:515] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.544632 18328 ts_tablet_manager.cc:1434] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.062s user 0.006s sys 0.013s
I20260317 16:17:15.546187 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.547374 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.561843 18326 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:15.562345 18326 ts_tablet_manager.cc:1403] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.111s user 0.032s sys 0.055s
I20260317 16:17:15.562933 18327 tablet_bootstrap.cc:654] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.563048 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:15.567420 18328 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:15.569319 18555 leader_election.cc:290] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:15.570044 18178 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 44678.7us
W20260317 16:17:15.571516 18175 leader_election.cc:343] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.572294 18660 raft_consensus.cc:493] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.578498 18660 raft_consensus.cc:515] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.579078 18326 raft_consensus.cc:359] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.579861 18326 raft_consensus.cc:385] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.580142 18326 raft_consensus.cc:740] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.581127 18660 leader_election.cc:290] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.580776 18326 consensus_queue.cc:260] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.584327 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2a036705e4a4da5a5392d119edb244e" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:15.585842 18175 leader_election.cc:343] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.589043 18328 tablet_bootstrap.cc:654] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.589226 18502 raft_consensus.cc:493] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.589823 18502 raft_consensus.cc:515] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.592083 18502 leader_election.cc:290] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.596804 18326 ts_tablet_manager.cc:1434] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260317 16:17:15.575443 18251 leader_election.cc:304] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.598043 18326 tablet_bootstrap.cc:492] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:15.599607 18502 raft_consensus.cc:2749] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.601795 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:15.613840 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:15.602176 18603 raft_consensus.cc:515] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.617800 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2a036705e4a4da5a5392d119edb244e" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.618916 18284 raft_consensus.cc:2468] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:15.628543 18662 raft_consensus.cc:493] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.629508 18327 tablet_bootstrap.cc:492] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.630076 18327 ts_tablet_manager.cc:1403] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.138s user 0.010s sys 0.013s
I20260317 16:17:15.629122 18662 raft_consensus.cc:515] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.633297 18327 raft_consensus.cc:359] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.634289 18327 raft_consensus.cc:385] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.636638 18178 leader_election.cc:304] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.636379 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:15.621927 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.634662 18327 raft_consensus.cc:740] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.640558 18600 raft_consensus.cc:2804] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:15.644464 18600 raft_consensus.cc:493] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:15.644853 18600 raft_consensus.cc:3060] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.646209 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c9d82ef74c442299f4a3cf04a5bb2d7" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:15.643071 18326 tablet_bootstrap.cc:654] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.647159 18207 raft_consensus.cc:3060] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.643899 18327 consensus_queue.cc:260] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:15.655973 18251 leader_election.cc:343] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.643326 18283 raft_consensus.cc:3060] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.656719 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:15.668103 18249 leader_election.cc:343] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.669001 18328 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:15.668886 18249 leader_election.cc:304] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.669515 18328 ts_tablet_manager.cc:1403] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.102s user 0.014s sys 0.022s
I20260317 16:17:15.671969 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c9d82ef74c442299f4a3cf04a5bb2d7" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:15.672561 18281 raft_consensus.cc:3060] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.678506 18327 ts_tablet_manager.cc:1434] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.048s user 0.005s sys 0.006s
I20260317 16:17:15.666129 18662 leader_election.cc:290] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.684139 18327 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:15.673738 18666 raft_consensus.cc:2749] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.686271 18283 raft_consensus.cc:2468] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
W20260317 16:17:15.690183 18178 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 50571.1us
I20260317 16:17:15.692339 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:15.693470 18283 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.670104 18603 leader_election.cc:290] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.708859 18207 raft_consensus.cc:2468] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:15.710218 18600 raft_consensus.cc:515] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.712538 18600 leader_election.cc:290] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:15.710673 18281 raft_consensus.cc:2468] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
W20260317 16:17:15.713290 18175 leader_election.cc:343] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.719656 18103 leader_election.cc:304] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:15.719981 18178 leader_election.cc:304] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.720664 18514 raft_consensus.cc:2804] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:15.721138 18514 raft_consensus.cc:697] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
W20260317 16:17:15.721810 18178 leader_election.cc:343] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.721983 18514 consensus_queue.cc:237] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.722219 18178 leader_election.cc:304] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:15.724980 18662 raft_consensus.cc:2804] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:15.725749 18328 raft_consensus.cc:359] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.725651 18662 raft_consensus.cc:697] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:15.727068 18556 raft_consensus.cc:2749] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:15.727523 18662 consensus_queue.cc:237] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.729001 18328 raft_consensus.cc:385] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.729382 18328 raft_consensus.cc:740] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.731272 18327 tablet_bootstrap.cc:654] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.731910 18326 tablet_bootstrap.cc:492] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:15.732349 18326 ts_tablet_manager.cc:1403] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.135s user 0.017s sys 0.012s
I20260317 16:17:15.730094 18328 consensus_queue.cc:260] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.736961 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2a036705e4a4da5a5392d119edb244e" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:15.737564 18326 raft_consensus.cc:359] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.738528 18326 raft_consensus.cc:385] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.739149 18326 raft_consensus.cc:740] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.740636 18326 consensus_queue.cc:260] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.737614 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2a036705e4a4da5a5392d119edb244e" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:15.743161 18136 raft_consensus.cc:3060] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.747102 18326 ts_tablet_manager.cc:1434] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.014s user 0.002s sys 0.005s
I20260317 16:17:15.749272 18326 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:15.746500 18281 raft_consensus.cc:3060] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.758319 18663 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:15.759940 18663 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:15.760383 18328 ts_tablet_manager.cc:1434] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.091s user 0.007s sys 0.002s
I20260317 16:17:15.763331 18136 raft_consensus.cc:2468] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:15.763808 18328 tablet_bootstrap.cc:492] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:15.767868 18175 leader_election.cc:304] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:15.769841 18135 raft_consensus.cc:3060] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:15.772188 18663 raft_consensus.cc:2804] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:15.795377 18327 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:15.796151 18327 ts_tablet_manager.cc:1403] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.112s user 0.009s sys 0.008s
I20260317 16:17:15.778576 18663 raft_consensus.cc:697] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:15.803804 18710 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:15.805745 18034 catalog_manager.cc:5671] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.806478 18035 catalog_manager.cc:5671] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.809147 18663 consensus_queue.cc:237] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.817255 18281 raft_consensus.cc:2468] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:15.829058 18326 tablet_bootstrap.cc:654] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.835096 18327 raft_consensus.cc:359] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.840723 18327 raft_consensus.cc:385] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.841003 18327 raft_consensus.cc:740] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.841640 18327 consensus_queue.cc:260] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.853626 18327 ts_tablet_manager.cc:1434] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.057s user 0.007s sys 0.003s
I20260317 16:17:15.854897 18327 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:15.901916 18328 tablet_bootstrap.cc:654] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.923081 18326 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:15.923835 18326 ts_tablet_manager.cc:1403] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.175s user 0.020s sys 0.022s
I20260317 16:17:15.930539 18327 tablet_bootstrap.cc:654] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.933678 18326 raft_consensus.cc:359] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.934407 18326 raft_consensus.cc:385] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:15.934624 18326 raft_consensus.cc:740] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:15.935220 18326 consensus_queue.cc:260] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.939841 18326 ts_tablet_manager.cc:1434] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260317 16:17:15.945560 18034 catalog_manager.cc:5671] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:15.947541 18666 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.948195 18666 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.950708 18666 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.958297 18326 tablet_bootstrap.cc:492] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:15.954082 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:15.974419 18662 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:15.982097 18326 tablet_bootstrap.cc:654] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:15.954144 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:15.986089 18417 raft_consensus.cc:493] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:15.987890 18210 raft_consensus.cc:2393] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:15.994263 18417 raft_consensus.cc:515] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:15.996290 18417 leader_election.cc:290] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:15.997078 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:15.998196 18249 leader_election.cc:343] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:15.998991 18135 raft_consensus.cc:2393] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 021d78130690452ba094f4a392fb4378 in current term 1: Already voted for candidate 94edf207909c4f8990bfe29ed1a87a6d in this term.
I20260317 16:17:16.001901 18327 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:16.002399 18327 ts_tablet_manager.cc:1403] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.148s user 0.012s sys 0.000s
I20260317 16:17:16.004866 18328 tablet_bootstrap.cc:492] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:16.005645 18328 ts_tablet_manager.cc:1403] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.242s user 0.013s sys 0.010s
I20260317 16:17:16.008215 18328 raft_consensus.cc:359] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.008859 18328 raft_consensus.cc:385] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.009131 18328 raft_consensus.cc:740] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.011394 18328 consensus_queue.cc:260] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.012526 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.013533 18211 raft_consensus.cc:2468] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:16.016867 18417 raft_consensus.cc:493] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.017391 18417 raft_consensus.cc:515] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.019716 18417 leader_election.cc:290] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:16.023140 18328 ts_tablet_manager.cc:1434] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260317 16:17:16.024387 18328 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:16.022655 18327 raft_consensus.cc:359] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.025218 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ed2db954354e669de7909d315142aa" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:16.025563 18327 raft_consensus.cc:385] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.025969 18327 raft_consensus.cc:740] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
W20260317 16:17:16.026418 18249 leader_election.cc:343] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.029556 18326 tablet_bootstrap.cc:492] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.030799 18328 tablet_bootstrap.cc:654] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.030084 18327 consensus_queue.cc:260] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.033205 18327 ts_tablet_manager.cc:1434] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.031s user 0.004s sys 0.004s
I20260317 16:17:16.035121 18327 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:16.033833 18326 ts_tablet_manager.cc:1403] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.073s user 0.023s sys 0.004s
I20260317 16:17:16.011667 18251 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.038349 18417 raft_consensus.cc:3060] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.039422 18326 raft_consensus.cc:359] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.040131 18326 raft_consensus.cc:385] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.041597 18326 raft_consensus.cc:740] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.042452 18326 consensus_queue.cc:260] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.046145 18327 tablet_bootstrap.cc:654] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.046655 18328 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:16.047231 18328 ts_tablet_manager.cc:1403] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.004s
I20260317 16:17:16.046674 18417 raft_consensus.cc:2749] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:16.053544 18605 raft_consensus.cc:493] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.053326 18328 raft_consensus.cc:359] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.054121 18328 raft_consensus.cc:385] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.054008 18605 raft_consensus.cc:515] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.055220 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ed2db954354e669de7909d315142aa" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.055434 18251 leader_election.cc:304] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.055779 18211 raft_consensus.cc:2468] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:16.056164 18665 raft_consensus.cc:2804] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.054383 18328 raft_consensus.cc:740] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.056596 18665 raft_consensus.cc:493] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.057189 18665 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.057761 18605 leader_election.cc:290] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.060775 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:16.065524 18286 raft_consensus.cc:2393] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:16.063900 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.067778 18102 leader_election.cc:304] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:16.068517 18417 raft_consensus.cc:493] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.069190 18602 raft_consensus.cc:3060] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.069003 18417 raft_consensus.cc:515] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.064703 18251 leader_election.cc:304] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.070420 18666 raft_consensus.cc:2804] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.057159 18328 consensus_queue.cc:260] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.070819 18666 raft_consensus.cc:493] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.071256 18666 raft_consensus.cc:3060] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.071867 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:16.072835 18328 ts_tablet_manager.cc:1434] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20260317 16:17:16.076619 18417 leader_election.cc:290] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:16.076984 18602 raft_consensus.cc:2749] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:16.084205 18328 tablet_bootstrap.cc:492] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:16.087067 18326 ts_tablet_manager.cc:1434] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
W20260317 16:17:16.087666 18249 leader_election.cc:343] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.088462 18326 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:16.095427 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
W20260317 16:17:16.099054 18251 leader_election.cc:343] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.099503 18251 leader_election.cc:304] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.100323 18728 raft_consensus.cc:2749] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:16.100898 18328 tablet_bootstrap.cc:654] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.106175 18665 raft_consensus.cc:515] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.085743 18666 raft_consensus.cc:515] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.111824 18666 leader_election.cc:290] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:16.113647 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:16.114040 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ed2db954354e669de7909d315142aa" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
W20260317 16:17:16.115520 18249 leader_election.cc:343] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:16.116298 18249 leader_election.cc:343] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.119313 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:16.120252 18211 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.123342 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ed2db954354e669de7909d315142aa" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:16.124190 18210 raft_consensus.cc:3060] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.127997 18328 tablet_bootstrap.cc:492] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:16.128535 18328 ts_tablet_manager.cc:1403] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.014s
I20260317 16:17:16.131726 18328 raft_consensus.cc:359] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.132679 18328 raft_consensus.cc:385] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.132993 18328 raft_consensus.cc:740] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.134441 18665 leader_election.cc:290] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:16.133914 18328 consensus_queue.cc:260] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.141484 18211 raft_consensus.cc:2468] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:16.142935 18251 leader_election.cc:304] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.143579 18210 raft_consensus.cc:2468] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:16.144393 18665 raft_consensus.cc:2804] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.145102 18665 raft_consensus.cc:697] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:16.145524 18327 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:16.146049 18327 ts_tablet_manager.cc:1403] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.111s user 0.008s sys 0.039s
I20260317 16:17:16.146040 18665 consensus_queue.cc:237] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.147209 18328 ts_tablet_manager.cc:1434] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.018s user 0.004s sys 0.005s
I20260317 16:17:16.148360 18328 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: Bootstrap starting.
I20260317 16:17:16.148602 18251 leader_election.cc:304] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.149420 18666 raft_consensus.cc:2804] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.149052 18327 raft_consensus.cc:359] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.149999 18327 raft_consensus.cc:385] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.150213 18666 raft_consensus.cc:697] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:16.150288 18327 raft_consensus.cc:740] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.151257 18327 consensus_queue.cc:260] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.151777 18666 consensus_queue.cc:237] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.154592 18327 ts_tablet_manager.cc:1434] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260317 16:17:16.155368 18326 tablet_bootstrap.cc:654] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.156455 18328 tablet_bootstrap.cc:654] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.155990 18327 tablet_bootstrap.cc:492] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:16.170668 18663 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:16.174034 18327 tablet_bootstrap.cc:654] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.189275 18660 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:16.182495 18034 catalog_manager.cc:5671] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:16.205467 18326 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.205928 18326 ts_tablet_manager.cc:1403] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.118s user 0.011s sys 0.002s
I20260317 16:17:16.223687 18326 raft_consensus.cc:359] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.224623 18326 raft_consensus.cc:385] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.224903 18326 raft_consensus.cc:740] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.225543 18326 consensus_queue.cc:260] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.241478 18605 raft_consensus.cc:493] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.241843 18660 consensus_queue.cc:1048] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.242431 18605 raft_consensus.cc:515] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.254558 18605 leader_election.cc:290] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.258733 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.259505 18211 raft_consensus.cc:2468] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:16.260892 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:16.252058 18603 raft_consensus.cc:493] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.264055 18603 raft_consensus.cc:515] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.265938 18603 leader_election.cc:290] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.272514 18663 consensus_queue.cc:1048] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.273963 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.274878 18207 raft_consensus.cc:2468] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:16.275794 18102 leader_election.cc:304] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.276304 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:16.276989 18281 raft_consensus.cc:2468] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 0.
I20260317 16:17:16.277801 18514 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 16:17:16.276566 18103 leader_election.cc:343] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.280609 18103 leader_election.cc:304] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.281088 18603 raft_consensus.cc:2804] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.281688 18603 raft_consensus.cc:493] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.282264 18603 raft_consensus.cc:3060] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.283811 18602 raft_consensus.cc:2804] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.284333 18602 raft_consensus.cc:493] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.284739 18602 raft_consensus.cc:3060] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.325553 18514 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:16.328137 18326 ts_tablet_manager.cc:1434] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.122s user 0.004s sys 0.004s
I20260317 16:17:16.340086 18326 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:16.344323 18602 raft_consensus.cc:515] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.346638 18602 leader_election.cc:290] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.347966 18603 raft_consensus.cc:515] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.352437 18328 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: No bootstrap required, opened a new log
I20260317 16:17:16.355046 18603 leader_election.cc:290] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.355547 18328 ts_tablet_manager.cc:1403] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: Time spent bootstrapping tablet: real 0.207s user 0.048s sys 0.070s
I20260317 16:17:16.364629 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:16.365290 18211 raft_consensus.cc:3060] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.371234 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:16.378286 18207 raft_consensus.cc:3060] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.396507 18211 raft_consensus.cc:2468] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:16.404551 18328 raft_consensus.cc:359] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.409142 18328 raft_consensus.cc:385] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.410794 18328 raft_consensus.cc:740] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.411581 18328 consensus_queue.cc:260] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.410038 18207 raft_consensus.cc:2468] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:16.415952 18328 ts_tablet_manager.cc:1434] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260317 16:17:16.417840 18102 leader_election.cc:304] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.419240 18603 raft_consensus.cc:2804] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.420311 18603 raft_consensus.cc:697] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:16.422320 18603 consensus_queue.cc:237] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:16.427537 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 123595us
I20260317 16:17:16.432868 18102 leader_election.cc:304] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.436574 18327 tablet_bootstrap.cc:492] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:16.459931 18327 ts_tablet_manager.cc:1403] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.304s user 0.056s sys 0.085s
I20260317 16:17:16.463372 18738 raft_consensus.cc:2804] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.464321 18738 raft_consensus.cc:697] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:16.448462 18326 tablet_bootstrap.cc:654] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.450295 18663 raft_consensus.cc:493] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.465620 18663 raft_consensus.cc:515] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.465948 18738 consensus_queue.cc:237] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.453090 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:16.466629 18327 raft_consensus.cc:359] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.455623 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:16.467393 18327 raft_consensus.cc:385] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.467702 18327 raft_consensus.cc:740] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.467025 18283 raft_consensus.cc:3060] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.468480 18327 consensus_queue.cc:260] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.469339 18281 raft_consensus.cc:3060] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.468506 18663 leader_election.cc:290] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.441167 18034 catalog_manager.cc:5671] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:16.487692 18284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:16.489277 18284 raft_consensus.cc:2468] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:16.483597 18281 raft_consensus.cc:2468] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:16.494719 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:16.488660 18283 raft_consensus.cc:2468] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:16.497680 18327 ts_tablet_manager.cc:1434] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.037s user 0.007s sys 0.001s
I20260317 16:17:16.500420 18178 leader_election.cc:304] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:16.501366 18744 raft_consensus.cc:2804] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.501765 18744 raft_consensus.cc:493] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.502094 18744 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.498653 18327 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: Bootstrap starting.
I20260317 16:17:16.514962 18326 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.516371 18744 raft_consensus.cc:515] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:16.516263 18175 leader_election.cc:343] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.516892 18326 ts_tablet_manager.cc:1403] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.177s user 0.013s sys 0.005s
I20260317 16:17:16.519690 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:16.520318 18281 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.520573 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:16.520939 18327 tablet_bootstrap.cc:654] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.523037 18032 catalog_manager.cc:5671] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:16.529390 18708 raft_consensus.cc:493] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.529965 18708 raft_consensus.cc:515] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.530550 18032 catalog_manager.cc:5671] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:16.534058 18249 consensus_peers.cc:597] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:16.525204 18744 leader_election.cc:290] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.534922 18708 leader_election.cc:290] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.536696 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:16.537372 18282 raft_consensus.cc:2468] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 0.
I20260317 16:17:16.538409 18178 leader_election.cc:304] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:16.539975 18708 raft_consensus.cc:2804] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.540364 18708 raft_consensus.cc:493] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.540746 18708 raft_consensus.cc:3060] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.540797 18281 raft_consensus.cc:2468] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:16.542281 18178 leader_election.cc:304] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:16.543354 18744 raft_consensus.cc:2804] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.543978 18744 raft_consensus.cc:697] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:16.545089 18744 consensus_queue.cc:237] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.579152 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
W20260317 16:17:16.583914 18175 leader_election.cc:343] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:17:16.585150 18175 leader_election.cc:343] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.587174 18326 raft_consensus.cc:359] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.587769 18326 raft_consensus.cc:385] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.588049 18326 raft_consensus.cc:740] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.593204 18326 consensus_queue.cc:260] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.599534 18326 ts_tablet_manager.cc:1434] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.081s user 0.006s sys 0.000s
I20260317 16:17:16.631006 18708 raft_consensus.cc:515] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.633718 18708 leader_election.cc:290] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.633809 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:16.634891 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
W20260317 16:17:16.635181 18175 leader_election.cc:343] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:17:16.635444 18282 raft_consensus.cc:3060] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.645895 18282 raft_consensus.cc:2468] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 1.
I20260317 16:17:16.650919 18178 leader_election.cc:304] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:16.654140 18556 raft_consensus.cc:2804] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.654659 18556 raft_consensus.cc:697] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:16.655587 18556 consensus_queue.cc:237] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.670202 18326 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:16.678874 18326 tablet_bootstrap.cc:654] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.682741 18327 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: No bootstrap required, opened a new log
I20260317 16:17:16.683305 18327 ts_tablet_manager.cc:1403] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: Time spent bootstrapping tablet: real 0.185s user 0.071s sys 0.087s
I20260317 16:17:16.686254 18327 raft_consensus.cc:359] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.687022 18327 raft_consensus.cc:385] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.687326 18327 raft_consensus.cc:740] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.688095 18327 consensus_queue.cc:260] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.691656 18665 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.695695 18034 catalog_manager.cc:5671] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:16.722406 18034 catalog_manager.cc:5671] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:16.726552 18083 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:17:16.729826 18327 ts_tablet_manager.cc:1434] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: Time spent starting tablet: real 0.046s user 0.005s sys 0.007s
I20260317 16:17:16.732792 18137 raft_consensus.cc:3060] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.746007 18728 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.746356 18326 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.750525 18326 ts_tablet_manager.cc:1403] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.012s
I20260317 16:17:16.753315 18326 raft_consensus.cc:359] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.754142 18326 raft_consensus.cc:385] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.754442 18326 raft_consensus.cc:740] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.756345 18326 consensus_queue.cc:260] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.758188 18326 ts_tablet_manager.cc:1434] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260317 16:17:16.759603 18326 tablet_bootstrap.cc:492] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:16.765179 18326 tablet_bootstrap.cc:654] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.783983 18666 consensus_queue.cc:1048] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.785663 18326 tablet_bootstrap.cc:492] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.786154 18326 ts_tablet_manager.cc:1403] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.007s
I20260317 16:17:16.789405 18326 raft_consensus.cc:359] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.790172 18326 raft_consensus.cc:385] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.790437 18326 raft_consensus.cc:740] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.791092 18326 consensus_queue.cc:260] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.794095 18326 ts_tablet_manager.cc:1434] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260317 16:17:16.795555 18326 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:16.802567 18326 tablet_bootstrap.cc:654] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.817147 18326 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.817638 18326 ts_tablet_manager.cc:1403] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20260317 16:17:16.820197 18326 raft_consensus.cc:359] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.820883 18326 raft_consensus.cc:385] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.821151 18326 raft_consensus.cc:740] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.821892 18326 consensus_queue.cc:260] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.824438 18326 ts_tablet_manager.cc:1434] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 16:17:16.825433 18326 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d: Bootstrap starting.
I20260317 16:17:16.832558 18326 tablet_bootstrap.cc:654] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d: Neither blocks nor log segments found. Creating new log.
I20260317 16:17:16.837523 18326 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d: No bootstrap required, opened a new log
I20260317 16:17:16.838085 18326 ts_tablet_manager.cc:1403] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260317 16:17:16.840463 18326 raft_consensus.cc:359] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.841194 18326 raft_consensus.cc:385] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:17:16.841615 18326 raft_consensus.cc:740] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Initialized, Role: FOLLOWER
I20260317 16:17:16.842270 18326 consensus_queue.cc:260] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.844234 18326 ts_tablet_manager.cc:1434] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260317 16:17:16.860358 18502 raft_consensus.cc:493] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.860885 18502 raft_consensus.cc:515] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.862515 18502 leader_election.cc:290] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:16.863448 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.863262 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d" is_pre_election: true
I20260317 16:17:16.864111 18210 raft_consensus.cc:2468] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:16.864285 18137 raft_consensus.cc:2468] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 0.
I20260317 16:17:16.866202 18251 leader_election.cc:304] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:16.867311 18502 raft_consensus.cc:2804] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:17:16.868516 18502 raft_consensus.cc:493] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.868885 18502 raft_consensus.cc:3060] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.874542 18502 raft_consensus.cc:515] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.877096 18502 leader_election.cc:290] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:16.878086 18133 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:16.878623 18133 raft_consensus.cc:3060] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.878363 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:16.879150 18207 raft_consensus.cc:3060] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.879814 18738 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.886291 18133 raft_consensus.cc:2468] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:16.886284 18207 raft_consensus.cc:2468] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:16.887648 18249 leader_election.cc:304] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.890894 18502 raft_consensus.cc:2804] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:17:16.891433 18502 raft_consensus.cc:697] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:16.892087 18738 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.892282 18502 consensus_queue.cc:237] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.901072 18603 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:16.901664 18603 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.907323 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:16.907967 18210 raft_consensus.cc:2468] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:16.909273 18102 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.910524 18740 raft_consensus.cc:2804] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:17:16.910900 18740 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:17:16.911257 18740 raft_consensus.cc:3060] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:16.917093 18603 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.923877 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:16.924744 18282 raft_consensus.cc:2468] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 1.
I20260317 16:17:16.932354 18740 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:16.937034 18603 consensus_queue.cc:1048] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.937418 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:16.938472 18210 raft_consensus.cc:3060] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:16.967279 18740 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:16.958273 18210 raft_consensus.cc:2468] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:16.970367 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d3017628c5244c3a8bb091d0a4768fd" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:16.971207 18282 raft_consensus.cc:3060] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:16.971243 18102 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:16.973146 18738 raft_consensus.cc:2804] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:16.979478 18603 consensus_queue.cc:1048] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:16.994284 18282 raft_consensus.cc:2468] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:16.994606 18137 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:16.996908 18034 catalog_manager.cc:5671] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:17.011030 18738 raft_consensus.cc:697] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:17.012022 18738 consensus_queue.cc:237] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:17.025177 18744 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:17.051615 18133 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:17.054536 18740 raft_consensus.cc:493] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:17:17.055094 18740 raft_consensus.cc:515] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:17.056979 18740 leader_election.cc:290] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:17.058419 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3a81306894b71bf2eacc88c770eaa" is_pre_election: true
I20260317 16:17:17.060808 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:17.061589 18281 raft_consensus.cc:2393] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94edf207909c4f8990bfe29ed1a87a6d in current term 1: Already voted for candidate 88b3a81306894b71bf2eacc88c770eaa in this term.
I20260317 16:17:17.067631 18103 leader_election.cc:304] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:17.068795 18738 raft_consensus.cc:3060] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:17:17.073765 18031 catalog_manager.cc:5671] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 0 to 2, leader changed from <none> to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:17.077193 18663 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:17.078653 18738 raft_consensus.cc:2749] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:17:17.130978 18663 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:17.144176 18663 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:17.171051 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:17:17.181517 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:17.183779 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:17:17.184885 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:17:17.218274 16879 server_base.cc:1060] running on GCE node
I20260317 16:17:17.220923 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:17:17.221246 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:17:17.221453 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764237221436 us; error 0 us; skew 500 ppm
I20260317 16:17:17.222213 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:17:17.228000 16879 webserver.cc:492] Webserver started at http://127.16.123.196:33921/ using document root <none> and password file <none>
I20260317 16:17:17.228760 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:17:17.229064 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:17:17.229475 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:17:17.230741 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-3-root/instance:
uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
format_stamp: "Formatted at 2026-03-17 16:17:17 on dist-test-slave-kqc6"
I20260317 16:17:17.235793 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260317 16:17:17.239598 18778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:17.240572 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260317 16:17:17.240957 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-3-root
uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
format_stamp: "Formatted at 2026-03-17 16:17:17 on dist-test-slave-kqc6"
I20260317 16:17:17.241314 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:17:17.277887 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:17:17.279438 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:17:17.281131 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:17:17.285014 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:17:17.285248 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:17.285619 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:17:17.285823 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:17:17.309167 18728 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:17.343339 18665 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:17.434479 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.196:40243
I20260317 16:17:17.434645 18842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.196:40243 every 8 connection(s)
I20260317 16:17:17.473032 16879 auto_leader_rebalancer-test.cc:417] add a tserver: 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:17.477607 18843 heartbeater.cc:344] Connected to a master server at 127.16.123.254:39965
I20260317 16:17:17.478161 18843 heartbeater.cc:461] Registering TS with master...
I20260317 16:17:17.479023 18843 heartbeater.cc:507] Master 127.16.123.254:39965 requested a full tablet report, sending...
I20260317 16:17:17.481740 18032 ts_manager.cc:194] Registered new tserver with Master: 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:17.483682 18032 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60752
I20260317 16:17:17.556149 18210 raft_consensus.cc:1275] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:17:17.557596 18740 consensus_queue.cc:1048] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:17:17.563272 18282 raft_consensus.cc:1275] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:17:17.568354 18740 consensus_queue.cc:1048] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:17:17.823901 18210 consensus_queue.cc:237] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:17.831362 18281 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:17.834113 18660 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:17.836540 18137 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:17.838297 18662 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:17.851488 18663 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:17.857151 18281 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:17.859429 18175 consensus_peers.cc:597] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8284701fdaf640daa689a2534914514c. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:17.859031 18136 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:17.878362 18034 catalog_manager.cc:5671] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:17.879948 18211 consensus_queue.cc:237] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:17.897609 18175 consensus_peers.cc:597] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07c5d59b5b404eec925d5c4681953edb. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:17.899550 18137 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:17.899550 18283 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:17.901598 18663 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:17.902778 18857 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:17.929981 18663 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:17.931097 18136 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:17.933063 18281 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:17.963940 18031 catalog_manager.cc:5671] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:17.981153 18281 consensus_queue.cc:237] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:17.989101 18137 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:17.989178 18210 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:17.990830 18502 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:17.994279 18502 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260317 16:17:18.016825 18249 consensus_peers.cc:597] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e145759d6de4334931c0039a97abdd8. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:18.033460 18665 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.035804 18210 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.039426 18131 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.066177 18210 consensus_queue.cc:237] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.066428 18034 catalog_manager.cc:5671] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.083798 18131 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:18.083863 18281 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:18.085721 18857 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:18.087836 18662 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260317 16:17:18.094118 18175 consensus_peers.cc:597] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b643cd863d14f36a3b9341312ef4770. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:18.106491 18857 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.114881 18281 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.126458 18131 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:18.144979 18034 catalog_manager.cc:5671] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:18.342798 18874 ts_tablet_manager.cc:933] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:18.342911 18873 ts_tablet_manager.cc:933] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:18.359709 18873 tablet_copy_client.cc:323] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:18.360777 18874 tablet_copy_client.cc:323] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:18.379223 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 8284701fdaf640daa689a2534914514c from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:18.379446 18221 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 07c5d59b5b404eec925d5c4681953edb from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:18.379969 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 8284701fdaf640daa689a2534914514c from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-8284701fdaf640daa689a2534914514c
I20260317 16:17:18.381565 18221 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 07c5d59b5b404eec925d5c4681953edb from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-07c5d59b5b404eec925d5c4681953edb
I20260317 16:17:18.391072 18221 tablet_copy_source_session.cc:215] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:18.394613 18222 tablet_copy_source_session.cc:215] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:18.396107 18874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c5d59b5b404eec925d5c4681953edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:18.398046 18873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8284701fdaf640daa689a2534914514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:18.411159 18874 tablet_copy_client.cc:806] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:18.411935 18874 tablet_copy_client.cc:670] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:18.413106 18873 tablet_copy_client.cc:806] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:18.413694 18873 tablet_copy_client.cc:670] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:18.418830 18874 tablet_copy_client.cc:538] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:18.424167 18873 tablet_copy_client.cc:538] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:18.439321 18874 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:18.440308 18873 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:18.471661 18874 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:18.473145 18874 tablet_bootstrap.cc:492] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:18.473862 18874 ts_tablet_manager.cc:1403] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.005s
I20260317 16:17:18.478159 18874 raft_consensus.cc:359] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.479161 18874 raft_consensus.cc:740] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:18.479532 18873 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:18.480479 18873 tablet_bootstrap.cc:492] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:18.479869 18874 consensus_queue.cc:260] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.481215 18873 ts_tablet_manager.cc:1403] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.012s
I20260317 16:17:18.483700 18873 raft_consensus.cc:359] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.484614 18873 raft_consensus.cc:740] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:18.485392 18873 consensus_queue.cc:260] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.486912 18843 heartbeater.cc:499] Master 127.16.123.254:39965 was elected leader, sending a full tablet report...
I20260317 16:17:18.510242 18881 ts_tablet_manager.cc:933] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:18.530490 18884 ts_tablet_manager.cc:933] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:18.535503 18873 ts_tablet_manager.cc:1434] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.054s user 0.022s sys 0.033s
I20260317 16:17:18.537027 18874 ts_tablet_manager.cc:1434] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.063s user 0.021s sys 0.040s
I20260317 16:17:18.537649 18881 tablet_copy_client.cc:323] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:18.538278 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-8284701fdaf640daa689a2534914514c received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:18.539084 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-8284701fdaf640daa689a2534914514c on tablet 8284701fdaf640daa689a2534914514c with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:18.549978 18884 tablet_copy_client.cc:323] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:18.550206 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-07c5d59b5b404eec925d5c4681953edb received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:18.551192 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-07c5d59b5b404eec925d5c4681953edb on tablet 07c5d59b5b404eec925d5c4681953edb with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:18.569270 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 6b643cd863d14f36a3b9341312ef4770 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:18.569828 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 6b643cd863d14f36a3b9341312ef4770 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-6b643cd863d14f36a3b9341312ef4770
I20260317 16:17:18.578452 18222 tablet_copy_source_session.cc:215] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:18.581884 18884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b643cd863d14f36a3b9341312ef4770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:18.597491 18296 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 9e145759d6de4334931c0039a97abdd8 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:18.598182 18296 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 9e145759d6de4334931c0039a97abdd8 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-9e145759d6de4334931c0039a97abdd8
I20260317 16:17:18.605592 18296 tablet_copy_source_session.cc:215] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:18.608551 18881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e145759d6de4334931c0039a97abdd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:18.617470 18884 tablet_copy_client.cc:806] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:18.618057 18884 tablet_copy_client.cc:670] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:18.622207 18884 tablet_copy_client.cc:538] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:18.629144 18881 tablet_copy_client.cc:806] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:18.629797 18881 tablet_copy_client.cc:670] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:18.634873 18884 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:18.635802 18881 tablet_copy_client.cc:538] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:18.653519 18881 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:18.667130 18884 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:18.668254 18884 tablet_bootstrap.cc:492] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:18.668917 18884 ts_tablet_manager.cc:1403] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.007s
I20260317 16:17:18.676693 18884 raft_consensus.cc:359] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.677629 18884 raft_consensus.cc:740] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:18.678589 18884 consensus_queue.cc:260] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.681273 18884 ts_tablet_manager.cc:1434] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260317 16:17:18.686331 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-6b643cd863d14f36a3b9341312ef4770 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:18.686802 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-6b643cd863d14f36a3b9341312ef4770 on tablet 6b643cd863d14f36a3b9341312ef4770 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:18.690438 18881 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:18.691344 18881 tablet_bootstrap.cc:492] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:18.691975 18881 ts_tablet_manager.cc:1403] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.023s
I20260317 16:17:18.695170 18881 raft_consensus.cc:359] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.695793 18881 raft_consensus.cc:740] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:18.696434 18881 consensus_queue.cc:260] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:18.698590 18881 ts_tablet_manager.cc:1434] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260317 16:17:18.701149 18296 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-9e145759d6de4334931c0039a97abdd8 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:18.701611 18296 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-9e145759d6de4334931c0039a97abdd8 on tablet 9e145759d6de4334931c0039a97abdd8 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:18.728418 18817 raft_consensus.cc:1217] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:18.902413 18817 raft_consensus.cc:1217] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:18.915537 18817 raft_consensus.cc:1217] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:19.070241 18817 raft_consensus.cc:1217] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:19.121858 18869 raft_consensus.cc:1064] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:19.126298 18869 consensus_queue.cc:237] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.134541 18283 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.135244 18817 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.135376 18137 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.136389 18869 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.148118 18662 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:19.151316 18662 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.168884 18869 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.179463 18817 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.185192 18131 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.188359 18283 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.197618 18032 catalog_manager.cc:5671] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.240489 18207 consensus_queue.cc:237] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.245434 18817 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.246789 18869 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:19.247483 18131 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.249315 18662 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:19.256513 18662 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.259255 18817 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.259310 18131 raft_consensus.cc:2955] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.271170 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 8284701fdaf640daa689a2534914514c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:19.290603 18034 catalog_manager.cc:5671] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.302336 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 8284701fdaf640daa689a2534914514c with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:19.317484 18900 tablet_replica.cc:333] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:19.319011 18900 raft_consensus.cc:2243] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:19.320242 18663 raft_consensus.cc:1064] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:19.321398 18900 raft_consensus.cc:2272] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:19.322762 18663 consensus_queue.cc:237] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.325081 18900 ts_tablet_manager.cc:1916] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:19.339563 18817 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.341084 18663 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.341750 18137 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.348394 18900 ts_tablet_manager.cc:1929] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:19.344383 18663 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260317 16:17:19.349398 18900 log.cc:1199] T 8284701fdaf640daa689a2534914514c P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/8284701fdaf640daa689a2534914514c
I20260317 16:17:19.351253 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 8284701fdaf640daa689a2534914514c (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:19.351301 18281 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.356436 18662 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.379056 18899 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.381192 18136 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.404856 18035 catalog_manager.cc:5671] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.414337 18869 raft_consensus.cc:1064] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:19.417878 18869 consensus_queue.cc:237] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.433228 18817 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.521673 18818 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.524428 18899 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.528332 18136 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.545192 18283 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.548810 18282 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.554265 18906 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:19.566110 18905 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.583475 18211 consensus_queue.cc:237] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.585774 18899 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.594611 18502 raft_consensus.cc:1064] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:19.596974 18502 consensus_queue.cc:237] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.599332 18281 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.602999 18818 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.605876 18136 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.606691 18906 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:19.618352 18817 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.622107 18663 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.623756 18284 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.629556 18905 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:19.632988 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 07c5d59b5b404eec925d5c4681953edb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:19.615674 18035 catalog_manager.cc:5671] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:19.640847 18817 raft_consensus.cc:2955] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.659107 18816 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.662374 18210 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.662621 18665 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.675408 18665 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.675854 18136 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:19.677695 18890 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:19.691035 18032 catalog_manager.cc:5671] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.698709 18283 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.720826 18210 consensus_queue.cc:237] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.732640 18817 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.734844 18131 raft_consensus.cc:1275] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.736568 18869 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:19.737627 18890 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.738307 18207 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.742914 18136 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.744513 18906 consensus_queue.cc:1048] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:19.753969 18906 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.766227 18031 catalog_manager.cc:5671] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.751031 18817 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.767426 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 07c5d59b5b404eec925d5c4681953edb with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:19.782677 18136 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.787433 18818 raft_consensus.cc:2955] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.790552 18916 tablet_replica.cc:333] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:19.790458 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 6b643cd863d14f36a3b9341312ef4770 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:19.791865 18916 raft_consensus.cc:2243] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:19.792613 18916 raft_consensus.cc:2272] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:19.802073 18916 ts_tablet_manager.cc:1916] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:19.804939 18032 catalog_manager.cc:5671] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:19.810721 18283 consensus_queue.cc:237] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:19.823477 18817 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.824821 18728 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:19.826377 18137 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:19.829329 18728 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:19.835637 18916 ts_tablet_manager.cc:1929] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:19.835013 18665 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.836102 18916 log.cc:1199] T 07c5d59b5b404eec925d5c4681953edb P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/07c5d59b5b404eec925d5c4681953edb
I20260317 16:17:19.839347 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 07c5d59b5b404eec925d5c4681953edb (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:19.838922 18817 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.853122 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9e145759d6de4334931c0039a97abdd8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:19.859344 18131 raft_consensus.cc:2955] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.863668 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 6b643cd863d14f36a3b9341312ef4770 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:19.872344 18034 catalog_manager.cc:5671] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:19.875965 18917 tablet_replica.cc:333] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:19.877003 18917 raft_consensus.cc:2243] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:19.877595 18917 raft_consensus.cc:2272] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:19.883142 18917 ts_tablet_manager.cc:1916] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:19.886338 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 9e145759d6de4334931c0039a97abdd8 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:19.906000 18918 tablet_replica.cc:333] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:19.907202 18918 raft_consensus.cc:2243] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:19.908264 18918 raft_consensus.cc:2272] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:19.911177 18917 ts_tablet_manager.cc:1929] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:19.911638 18917 log.cc:1199] T 6b643cd863d14f36a3b9341312ef4770 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/6b643cd863d14f36a3b9341312ef4770
I20260317 16:17:19.913836 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 6b643cd863d14f36a3b9341312ef4770 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:19.914358 18918 ts_tablet_manager.cc:1916] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:19.930874 18918 ts_tablet_manager.cc:1929] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:19.931255 18918 log.cc:1199] T 9e145759d6de4334931c0039a97abdd8 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/9e145759d6de4334931c0039a97abdd8
I20260317 16:17:19.932894 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 9e145759d6de4334931c0039a97abdd8 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:20.952575 18281 consensus_queue.cc:237] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:20.961874 18211 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:20.962592 18249 consensus_peers.cc:597] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe3f3aabd9b444868c9732a8e5fa7440. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:20.964594 18665 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:20.965080 18136 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:20.967988 18665 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:20.975356 18890 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:20.976878 18211 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:20.982877 18136 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:20.991285 18281 consensus_queue.cc:237] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:20.997083 18035 catalog_manager.cc:5671] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:21.005487 18249 consensus_peers.cc:597] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24063e0e6d6a4826899168d2524d5abc. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:21.011408 18137 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:21.012279 18211 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:21.014292 18890 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:21.015210 18665 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:21.038077 18890 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.041086 18131 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.056334 18206 consensus_queue.cc:237] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.062580 18211 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.088302 18032 catalog_manager.cc:5671] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:21.119570 18131 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:21.119663 18281 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:21.120208 18175 consensus_peers.cc:597] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f77a63cdb5604ab493b7f5f9dc921763. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:21.121029 18906 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:21.121950 18869 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:21.133641 18906 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.136032 18137 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.139523 18281 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.160331 18031 catalog_manager.cc:5671] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:21.163262 18283 consensus_queue.cc:237] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:21.173579 18249 consensus_peers.cc:597] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84ded5b9676242a18139f88dc63af2b4. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:21.176894 18137 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:21.178802 18890 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:21.181061 18206 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:21.182612 18890 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260317 16:17:21.195410 18930 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.197932 18137 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.211535 18206 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.219913 18032 catalog_manager.cc:5671] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:21.388582 18938 ts_tablet_manager.cc:933] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:21.390885 18938 tablet_copy_client.cc:323] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:21.392572 18296 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 24063e0e6d6a4826899168d2524d5abc from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:21.393146 18296 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 24063e0e6d6a4826899168d2524d5abc from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-24063e0e6d6a4826899168d2524d5abc
I20260317 16:17:21.406813 18296 tablet_copy_source_session.cc:215] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:21.412303 18938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24063e0e6d6a4826899168d2524d5abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:21.423705 18940 ts_tablet_manager.cc:933] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:21.438892 18938 tablet_copy_client.cc:806] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:21.439471 18938 tablet_copy_client.cc:670] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:21.440559 18940 tablet_copy_client.cc:323] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:21.443527 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet fe3f3aabd9b444868c9732a8e5fa7440 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:21.444141 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet fe3f3aabd9b444868c9732a8e5fa7440 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-fe3f3aabd9b444868c9732a8e5fa7440
I20260317 16:17:21.447019 18938 tablet_copy_client.cc:538] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:21.454402 18295 tablet_copy_source_session.cc:215] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:21.457087 18938 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:21.459201 18940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3f3aabd9b444868c9732a8e5fa7440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:21.483385 18940 tablet_copy_client.cc:806] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:21.484016 18940 tablet_copy_client.cc:670] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:21.487905 18940 tablet_copy_client.cc:538] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:21.491209 18938 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:21.494675 18938 tablet_bootstrap.cc:492] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:21.495718 18938 ts_tablet_manager.cc:1403] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.020s
I20260317 16:17:21.501854 18938 raft_consensus.cc:359] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.503049 18938 raft_consensus.cc:740] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:21.504674 18938 consensus_queue.cc:260] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.511224 18940 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:21.519776 18938 ts_tablet_manager.cc:1434] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.024s user 0.011s sys 0.000s
I20260317 16:17:21.522890 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-24063e0e6d6a4826899168d2524d5abc received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:21.523347 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-24063e0e6d6a4826899168d2524d5abc on tablet 24063e0e6d6a4826899168d2524d5abc with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:21.538200 18940 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:21.540040 18940 tablet_bootstrap.cc:492] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:21.541065 18940 ts_tablet_manager.cc:1403] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.011s
I20260317 16:17:21.614975 18940 raft_consensus.cc:359] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.617110 18940 raft_consensus.cc:740] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:21.617784 18940 consensus_queue.cc:260] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.622258 18940 ts_tablet_manager.cc:1434] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.081s user 0.010s sys 0.005s
I20260317 16:17:21.626754 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-fe3f3aabd9b444868c9732a8e5fa7440 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:21.627632 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-fe3f3aabd9b444868c9732a8e5fa7440 on tablet fe3f3aabd9b444868c9732a8e5fa7440 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:21.712486 18940 ts_tablet_manager.cc:933] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:21.736323 18940 tablet_copy_client.cc:323] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:21.740790 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet f77a63cdb5604ab493b7f5f9dc921763 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:21.741853 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet f77a63cdb5604ab493b7f5f9dc921763 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-f77a63cdb5604ab493b7f5f9dc921763
I20260317 16:17:21.765189 18222 tablet_copy_source_session.cc:215] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:21.769826 18940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77a63cdb5604ab493b7f5f9dc921763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:21.796013 18938 ts_tablet_manager.cc:933] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:21.798064 18940 tablet_copy_client.cc:806] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:21.799121 18940 tablet_copy_client.cc:670] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:21.799438 18938 tablet_copy_client.cc:323] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:21.801594 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 84ded5b9676242a18139f88dc63af2b4 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:21.802157 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 84ded5b9676242a18139f88dc63af2b4 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-84ded5b9676242a18139f88dc63af2b4
I20260317 16:17:21.806900 18940 tablet_copy_client.cc:538] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:21.823222 18940 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:21.869673 18295 tablet_copy_source_session.cc:215] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:21.874429 18938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ded5b9676242a18139f88dc63af2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:21.887563 18940 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:21.888747 18940 tablet_bootstrap.cc:492] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:21.889489 18940 ts_tablet_manager.cc:1403] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.067s user 0.042s sys 0.017s
I20260317 16:17:21.894878 18940 raft_consensus.cc:359] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.895936 18940 raft_consensus.cc:740] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:21.897004 18818 raft_consensus.cc:1217] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:21.897174 18940 consensus_queue.cc:260] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.923398 18938 tablet_copy_client.cc:806] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:21.928438 18938 tablet_copy_client.cc:670] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:21.937153 18818 raft_consensus.cc:1217] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:21.942286 18938 tablet_copy_client.cc:538] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:21.948163 18940 ts_tablet_manager.cc:1434] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.058s user 0.019s sys 0.035s
I20260317 16:17:21.951047 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-f77a63cdb5604ab493b7f5f9dc921763 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:21.951534 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-f77a63cdb5604ab493b7f5f9dc921763 on tablet f77a63cdb5604ab493b7f5f9dc921763 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:21.960089 18938 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:21.984167 18938 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:21.985877 18938 tablet_bootstrap.cc:492] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:21.986680 18938 ts_tablet_manager.cc:1403] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.005s
I20260317 16:17:21.989678 18938 raft_consensus.cc:359] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.990690 18938 raft_consensus.cc:740] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:21.991571 18938 consensus_queue.cc:260] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:21.994879 18938 ts_tablet_manager.cc:1434] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260317 16:17:21.999365 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-84ded5b9676242a18139f88dc63af2b4 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:21.999778 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-84ded5b9676242a18139f88dc63af2b4 on tablet 84ded5b9676242a18139f88dc63af2b4 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:22.185966 18818 raft_consensus.cc:1217] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:22.246454 18818 raft_consensus.cc:1217] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:22.330369 18890 raft_consensus.cc:1064] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:22.332723 18890 consensus_queue.cc:237] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.339867 18818 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.340726 18210 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.340817 18137 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.341686 18890 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:22.342993 18502 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.343849 18931 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:22.367009 18502 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.370645 18206 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.385015 18931 raft_consensus.cc:1064] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:22.388610 18931 consensus_queue.cc:237] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.390851 18818 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.394994 18131 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.405153 18031 catalog_manager.cc:5671] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.432236 18818 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.435259 18131 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.435185 18931 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:22.437618 18931 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.435259 18210 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.441493 18931 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.450799 18502 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.454253 18131 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.458263 18210 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.466241 18818 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.485181 18282 consensus_queue.cc:237] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.493651 18818 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:22.495961 18931 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:22.496260 18211 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:22.498921 18032 catalog_manager.cc:5671] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.505575 18502 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:22.509147 18890 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.511440 18817 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.523370 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 24063e0e6d6a4826899168d2524d5abc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:22.531575 18210 raft_consensus.cc:2955] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.542145 18281 consensus_queue.cc:237] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.551923 18817 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:22.553599 18958 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:22.559741 18137 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:22.562132 18031 catalog_manager.cc:5671] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.562433 18931 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.582567 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet fe3f3aabd9b444868c9732a8e5fa7440 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:22.586272 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 24063e0e6d6a4826899168d2524d5abc with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:22.587047 18502 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:22.585964 18817 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.613065 18967 tablet_replica.cc:333] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:22.614624 18967 raft_consensus.cc:2243] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:22.616331 18967 raft_consensus.cc:2272] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:22.612888 18137 raft_consensus.cc:2955] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.623594 18967 ts_tablet_manager.cc:1916] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:22.626147 18035 catalog_manager.cc:5671] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:22.646026 18967 ts_tablet_manager.cc:1929] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:22.647169 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet fe3f3aabd9b444868c9732a8e5fa7440 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:22.646592 18967 log.cc:1199] T 24063e0e6d6a4826899168d2524d5abc P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/24063e0e6d6a4826899168d2524d5abc
I20260317 16:17:22.656455 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 24063e0e6d6a4826899168d2524d5abc (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:22.661162 18968 tablet_replica.cc:333] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:22.662351 18968 raft_consensus.cc:2243] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:22.663201 18968 raft_consensus.cc:2272] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:22.670696 18968 ts_tablet_manager.cc:1916] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:22.702917 18968 ts_tablet_manager.cc:1929] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:22.703354 18968 log.cc:1199] T fe3f3aabd9b444868c9732a8e5fa7440 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/fe3f3aabd9b444868c9732a8e5fa7440
I20260317 16:17:22.710391 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet fe3f3aabd9b444868c9732a8e5fa7440 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:22.762658 18958 raft_consensus.cc:1064] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:22.766278 18958 consensus_queue.cc:237] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.776306 18817 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.777819 18502 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.778280 18211 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.779629 18131 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.782181 18502 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.785064 18502 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260317 16:17:22.800910 18502 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.802539 18210 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.807443 18136 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.814309 18817 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.828884 18034 catalog_manager.cc:5671] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:22.845906 18663 raft_consensus.cc:1064] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:22.848277 18663 consensus_queue.cc:237] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.870038 18282 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.870546 18136 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.873309 18663 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.872493 18284 consensus_queue.cc:237] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:22.874338 18869 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:22.879642 18818 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:22.881292 18931 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:22.889497 18211 raft_consensus.cc:1275] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:22.891021 18931 consensus_queue.cc:1048] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:22.893349 18817 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:22.893433 18890 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.899959 18906 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:22.905066 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 84ded5b9676242a18139f88dc63af2b4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:22.906046 18817 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.917630 18211 raft_consensus.cc:2955] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.925026 18034 catalog_manager.cc:5671] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:22.942806 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 84ded5b9676242a18139f88dc63af2b4 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:22.943859 18967 tablet_replica.cc:333] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:22.945117 18967 raft_consensus.cc:2243] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:22.947589 18967 raft_consensus.cc:2272] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:22.952311 18967 ts_tablet_manager.cc:1916] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:22.965519 18136 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.970412 18979 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.972610 18818 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:22.987946 18282 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:23.001081 18034 catalog_manager.cc:5671] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:23.003549 18967 ts_tablet_manager.cc:1929] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:23.004037 18967 log.cc:1199] T 84ded5b9676242a18139f88dc63af2b4 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/84ded5b9676242a18139f88dc63af2b4
I20260317 16:17:23.005784 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 84ded5b9676242a18139f88dc63af2b4 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:23.029201 18211 consensus_queue.cc:237] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:23.037101 18818 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:23.038100 18136 raft_consensus.cc:1275] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:23.039724 18979 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:23.042029 18947 consensus_queue.cc:1048] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:23.050571 18663 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:23.052716 18131 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:23.056159 18818 raft_consensus.cc:2955] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:23.060969 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet f77a63cdb5604ab493b7f5f9dc921763 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:23.071404 18034 catalog_manager.cc:5671] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:23.090483 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet f77a63cdb5604ab493b7f5f9dc921763 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:23.092847 18981 tablet_replica.cc:333] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:23.094241 18981 raft_consensus.cc:2243] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:23.095081 18981 raft_consensus.cc:2272] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:23.097851 18981 ts_tablet_manager.cc:1916] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:23.115278 18981 ts_tablet_manager.cc:1929] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:23.115736 18981 log.cc:1199] T f77a63cdb5604ab493b7f5f9dc921763 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/f77a63cdb5604ab493b7f5f9dc921763
I20260317 16:17:23.117705 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet f77a63cdb5604ab493b7f5f9dc921763 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:24.162034 18137 consensus_queue.cc:237] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.171201 18210 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:24.171564 18283 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:24.173108 18945 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:24.175141 18969 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:24.205520 18969 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.207343 18206 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:24.210233 18100 consensus_peers.cc:597] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17fe426def0d4f378a44ce5ec328f2e0. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:24.209256 18282 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.230386 18283 consensus_queue.cc:237] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.231609 18031 catalog_manager.cc:5671] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:24.249243 18249 consensus_peers.cc:597] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55ecf327ed214b02bef87b8c05d54c26. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:24.252738 18136 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:24.252961 18206 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:24.254523 18931 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:24.256809 18931 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:24.288550 18993 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.293709 18136 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.328851 18206 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.330380 18211 consensus_queue.cc:237] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.349871 18034 catalog_manager.cc:5671] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.359299 18283 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:24.360774 19001 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:24.362279 18136 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:24.371281 18998 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260317 16:17:24.378407 18175 consensus_peers.cc:597] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09f47966c253483393878fddf97ac206. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:24.384081 19001 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.385828 18283 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.387822 18136 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.399144 18284 consensus_queue.cc:237] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:24.406056 18249 consensus_peers.cc:597] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 898420a8ccf84bbdba91710ca9c08ef1. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:24.406796 18032 catalog_manager.cc:5671] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.408550 18211 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:24.410081 18137 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:24.410172 18931 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:24.411547 18999 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:24.421742 18999 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.424232 18211 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.427691 18137 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:24.449237 18031 catalog_manager.cc:5671] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:24.762614 19007 ts_tablet_manager.cc:933] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:24.771950 19007 tablet_copy_client.cc:323] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:24.797235 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 17fe426def0d4f378a44ce5ec328f2e0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:24.797869 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 17fe426def0d4f378a44ce5ec328f2e0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-17fe426def0d4f378a44ce5ec328f2e0
I20260317 16:17:24.807667 18147 tablet_copy_source_session.cc:215] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:24.811107 19007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fe426def0d4f378a44ce5ec328f2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:24.818593 19011 ts_tablet_manager.cc:933] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:24.827214 19011 tablet_copy_client.cc:323] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:24.828840 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 09f47966c253483393878fddf97ac206 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:24.829411 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 09f47966c253483393878fddf97ac206 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-09f47966c253483393878fddf97ac206
I20260317 16:17:24.830492 19007 tablet_copy_client.cc:806] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:24.831434 19007 tablet_copy_client.cc:670] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:24.841178 18222 tablet_copy_source_session.cc:215] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:24.841773 19007 tablet_copy_client.cc:538] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:24.847437 19013 ts_tablet_manager.cc:933] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:24.852792 19007 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:24.852865 19011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f47966c253483393878fddf97ac206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:24.858165 19013 tablet_copy_client.cc:323] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:24.860687 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 55ecf327ed214b02bef87b8c05d54c26 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:24.861286 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 55ecf327ed214b02bef87b8c05d54c26 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-55ecf327ed214b02bef87b8c05d54c26
I20260317 16:17:24.876163 18295 tablet_copy_source_session.cc:215] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:24.885339 19013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ecf327ed214b02bef87b8c05d54c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:24.878968 19011 tablet_copy_client.cc:806] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:24.889076 19011 tablet_copy_client.cc:670] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:24.889989 19007 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:24.893528 19007 tablet_bootstrap.cc:492] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:24.895015 19007 ts_tablet_manager.cc:1403] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.004s
I20260317 16:17:24.897462 19011 tablet_copy_client.cc:538] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:24.902323 19007 raft_consensus.cc:359] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.903712 19007 raft_consensus.cc:740] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:24.904479 19007 consensus_queue.cc:260] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.915722 19007 ts_tablet_manager.cc:1434] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.020s user 0.011s sys 0.003s
I20260317 16:17:24.920001 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-17fe426def0d4f378a44ce5ec328f2e0 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:24.920475 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-17fe426def0d4f378a44ce5ec328f2e0 on tablet 17fe426def0d4f378a44ce5ec328f2e0 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:24.929581 19011 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:24.933204 19013 tablet_copy_client.cc:806] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:24.933979 19013 tablet_copy_client.cc:670] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:24.938822 19013 tablet_copy_client.cc:538] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:24.950038 19013 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:24.955667 19011 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:24.956882 19011 tablet_bootstrap.cc:492] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:24.957588 19011 ts_tablet_manager.cc:1403] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.007s
I20260317 16:17:24.960817 19011 raft_consensus.cc:359] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.961719 19011 raft_consensus.cc:740] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:24.962405 19011 consensus_queue.cc:260] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.964833 19011 ts_tablet_manager.cc:1434] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:17:24.966826 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-09f47966c253483393878fddf97ac206 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:24.967262 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-09f47966c253483393878fddf97ac206 on tablet 09f47966c253483393878fddf97ac206 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:24.984151 19013 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:24.985095 19013 tablet_bootstrap.cc:492] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:24.985863 19013 ts_tablet_manager.cc:1403] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.004s
I20260317 16:17:24.988682 19013 raft_consensus.cc:359] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.989621 19013 raft_consensus.cc:740] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:24.990188 19013 consensus_queue.cc:260] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:24.992518 19013 ts_tablet_manager.cc:1434] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260317 16:17:24.995489 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-55ecf327ed214b02bef87b8c05d54c26 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:24.995935 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-55ecf327ed214b02bef87b8c05d54c26 on tablet 55ecf327ed214b02bef87b8c05d54c26 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:25.005602 19013 ts_tablet_manager.cc:933] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:25.012439 19013 tablet_copy_client.cc:323] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:25.014075 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 898420a8ccf84bbdba91710ca9c08ef1 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:25.014659 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 898420a8ccf84bbdba91710ca9c08ef1 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-898420a8ccf84bbdba91710ca9c08ef1
I20260317 16:17:25.024758 18295 tablet_copy_source_session.cc:215] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:25.035828 19013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898420a8ccf84bbdba91710ca9c08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:25.056313 19013 tablet_copy_client.cc:806] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:25.057005 19013 tablet_copy_client.cc:670] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:25.062265 19013 tablet_copy_client.cc:538] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:25.070902 19013 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:25.092653 19013 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:25.093858 19013 tablet_bootstrap.cc:492] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:25.095221 19013 ts_tablet_manager.cc:1403] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.011s
I20260317 16:17:25.099336 19013 raft_consensus.cc:359] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:25.100541 19013 raft_consensus.cc:740] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:25.101467 19013 consensus_queue.cc:260] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:25.105311 19013 ts_tablet_manager.cc:1434] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260317 16:17:25.115331 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-898420a8ccf84bbdba91710ca9c08ef1 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:25.115815 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-898420a8ccf84bbdba91710ca9c08ef1 on tablet 898420a8ccf84bbdba91710ca9c08ef1 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:25.313611 18818 raft_consensus.cc:1217] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:25.349504 18818 raft_consensus.cc:1217] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260317 16:17:25.447530 18818 raft_consensus.cc:1217] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:25.569327 18818 raft_consensus.cc:1217] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:25.722622 18969 raft_consensus.cc:1064] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:25.725533 18969 consensus_queue.cc:237] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:25.739301 18818 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:25.740083 18206 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:25.740806 18282 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:25.740747 18989 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:25.741967 18969 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:25.744112 19020 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:25.767978 19020 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.774467 18282 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.779026 18206 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.799592 18818 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.803927 18034 catalog_manager.cc:5671] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.854310 19022 raft_consensus.cc:1064] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:25.860630 19022 consensus_queue.cc:237] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:25.878356 18818 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:25.878876 18135 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:25.882450 18931 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:25.885381 18931 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:25.895926 18206 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:25.894177 18135 consensus_queue.cc:237] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:25.899068 18931 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:25.920114 18817 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:25.921813 19020 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:25.925062 18282 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:25.926476 19020 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:25.931082 19026 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.932129 18931 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.937405 18818 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.946465 18817 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.949755 18136 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.950490 18206 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.953235 19022 raft_consensus.cc:1064] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:25.953665 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 17fe426def0d4f378a44ce5ec328f2e0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:25.957331 19022 consensus_queue.cc:237] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:25.958032 18035 catalog_manager.cc:5671] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.959959 19023 raft_consensus.cc:1064] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:25.965003 19023 consensus_queue.cc:237] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:25.982300 18282 raft_consensus.cc:2955] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:25.989488 18283 consensus_queue.cc:237] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:25.992995 18035 catalog_manager.cc:5671] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:26.020071 18818 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:26.020900 18816 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:26.025566 18211 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:26.028167 18135 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:26.037084 18999 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:26.043493 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 17fe426def0d4f378a44ce5ec328f2e0 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:26.048264 18136 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:26.063635 18663 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:26.065003 18816 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260317 16:17:26.074573 18251 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 53308.6us
I20260317 16:17:26.078048 19024 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:26.078126 18284 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:26.084834 19043 tablet_replica.cc:333] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:26.100450 18663 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:26.090539 19043 raft_consensus.cc:2243] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:26.115420 18979 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:26.116302 19043 raft_consensus.cc:2272] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:26.115617 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 84046us
I20260317 16:17:26.136101 18135 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:26.151650 19043 ts_tablet_manager.cc:1916] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:26.180696 19024 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:26.181849 19041 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:26.197342 18979 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.211583 18135 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.214661 19041 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:26.222594 19040 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.229250 18032 catalog_manager.cc:5671] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:26.213762 18816 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.235664 18818 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.243440 19043 ts_tablet_manager.cc:1929] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:26.262878 19043 log.cc:1199] T 17fe426def0d4f378a44ce5ec328f2e0 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/17fe426def0d4f378a44ce5ec328f2e0
I20260317 16:17:26.268784 18035 catalog_manager.cc:5671] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.278702 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 17fe426def0d4f378a44ce5ec328f2e0 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:26.278026 18206 consensus_queue.cc:237] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:26.284482 18284 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.287047 18210 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.294801 19041 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.316540 18817 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:26.317698 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 55ecf327ed214b02bef87b8c05d54c26 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:26.321020 19023 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:26.323292 18135 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:26.331761 19023 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.337025 18663 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:26.343705 18816 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.347424 18135 raft_consensus.cc:2955] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.349737 18817 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.355031 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 09f47966c253483393878fddf97ac206 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:26.361449 18136 raft_consensus.cc:2955] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.361254 18137 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.388386 18032 catalog_manager.cc:5671] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:26.386577 18031 catalog_manager.cc:5671] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.404302 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 09f47966c253483393878fddf97ac206 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:26.424975 18284 consensus_queue.cc:237] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:26.431097 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 55ecf327ed214b02bef87b8c05d54c26 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:26.432128 19043 tablet_replica.cc:333] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:26.433190 19043 raft_consensus.cc:2243] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:26.433539 19050 tablet_replica.cc:333] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:26.433950 19043 raft_consensus.cc:2272] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:26.437844 19050 raft_consensus.cc:2243] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:26.437992 19043 ts_tablet_manager.cc:1916] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:26.439527 19050 raft_consensus.cc:2272] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:26.451552 18816 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:26.454890 19022 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:26.463085 19043 ts_tablet_manager.cc:1929] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:26.464248 19043 log.cc:1199] T 55ecf327ed214b02bef87b8c05d54c26 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/55ecf327ed214b02bef87b8c05d54c26
I20260317 16:17:26.466548 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 55ecf327ed214b02bef87b8c05d54c26 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:26.472527 19041 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.476332 19050 ts_tablet_manager.cc:1916] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:26.480080 18816 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.488081 18210 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:26.490846 18931 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:26.500142 19050 ts_tablet_manager.cc:1929] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260317 16:17:26.500662 19050 log.cc:1199] T 09f47966c253483393878fddf97ac206 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/09f47966c253483393878fddf97ac206
I20260317 16:17:26.502571 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 09f47966c253483393878fddf97ac206 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:26.509521 18210 raft_consensus.cc:2955] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.513833 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 898420a8ccf84bbdba91710ca9c08ef1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:26.515766 18032 catalog_manager.cc:5671] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:26.555852 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 898420a8ccf84bbdba91710ca9c08ef1 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:26.563746 19051 tablet_replica.cc:333] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:26.566999 19051 raft_consensus.cc:2243] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:26.567782 19051 raft_consensus.cc:2272] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:26.572209 19051 ts_tablet_manager.cc:1916] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:26.592665 19051 ts_tablet_manager.cc:1929] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:26.593236 19051 log.cc:1199] T 898420a8ccf84bbdba91710ca9c08ef1 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/898420a8ccf84bbdba91710ca9c08ef1
I20260317 16:17:26.595063 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 898420a8ccf84bbdba91710ca9c08ef1 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:27.593063 18281 consensus_queue.cc:237] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:27.601328 18249 consensus_peers.cc:597] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9844da16b03545c5a9e5534b29b4343c. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:27.604530 18210 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:27.605573 18137 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:27.607033 19041 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:27.608071 19052 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:27.632513 19052 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.640587 18210 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.648229 18137 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.668704 18210 consensus_queue.cc:237] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:27.685562 18281 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:27.691672 18135 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:27.702006 19023 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:27.712353 18031 catalog_manager.cc:5671] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:27.719081 18175 consensus_peers.cc:597] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47801c6cdf6f461181a89246716390d7. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:27.720386 19001 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:27.736634 18281 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.735124 19023 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.778082 18136 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.790001 18211 consensus_queue.cc:237] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:27.798494 18281 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:27.798815 18175 consensus_peers.cc:597] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b451ac456d424629ae5ce6040cb15bd2. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:27.800678 19064 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:27.801908 18136 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:27.803622 19001 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:27.814143 19023 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.824806 18135 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.828424 18031 catalog_manager.cc:5671] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.843382 18284 consensus_queue.cc:237] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:27.855944 18283 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.859318 18034 catalog_manager.cc:5671] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.868109 18211 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:27.868124 18249 consensus_peers.cc:597] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6aca2493a994ed888cd663ddcf97582. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:27.871470 19041 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:27.872956 18135 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:27.874837 19056 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:27.893025 19041 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.901111 18135 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.898717 18211 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:27.927358 18034 catalog_manager.cc:5671] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:28.196457 19073 ts_tablet_manager.cc:933] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:28.198609 19073 tablet_copy_client.cc:323] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:28.200125 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 9844da16b03545c5a9e5534b29b4343c from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:28.200660 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 9844da16b03545c5a9e5534b29b4343c from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-9844da16b03545c5a9e5534b29b4343c
I20260317 16:17:28.206604 18295 tablet_copy_source_session.cc:215] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:28.209228 19073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9844da16b03545c5a9e5534b29b4343c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:28.225898 19073 tablet_copy_client.cc:806] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:28.226847 19073 tablet_copy_client.cc:670] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:28.230629 19073 tablet_copy_client.cc:538] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:28.239367 19073 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:28.262290 19073 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:28.263387 19073 tablet_bootstrap.cc:492] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:28.264082 19073 ts_tablet_manager.cc:1403] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20260317 16:17:28.266654 19073 raft_consensus.cc:359] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.267602 19073 raft_consensus.cc:740] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:28.268343 19073 consensus_queue.cc:260] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.274204 19073 ts_tablet_manager.cc:1434] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260317 16:17:28.277410 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-9844da16b03545c5a9e5534b29b4343c received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:28.277838 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-9844da16b03545c5a9e5534b29b4343c on tablet 9844da16b03545c5a9e5534b29b4343c with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:28.286649 19073 ts_tablet_manager.cc:933] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:28.288424 19073 tablet_copy_client.cc:323] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:28.290107 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 47801c6cdf6f461181a89246716390d7 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:28.290621 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 47801c6cdf6f461181a89246716390d7 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-47801c6cdf6f461181a89246716390d7
I20260317 16:17:28.298228 18222 tablet_copy_source_session.cc:215] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:28.302033 19073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47801c6cdf6f461181a89246716390d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:28.316947 19073 tablet_copy_client.cc:806] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:28.317596 19073 tablet_copy_client.cc:670] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:28.327659 19073 tablet_copy_client.cc:538] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:28.336560 19073 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:28.355458 19073 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:28.356317 19073 tablet_bootstrap.cc:492] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:28.356979 19073 ts_tablet_manager.cc:1403] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.003s
I20260317 16:17:28.360066 19073 raft_consensus.cc:359] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.360797 19073 raft_consensus.cc:740] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:28.361545 19073 consensus_queue.cc:260] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.364259 19073 ts_tablet_manager.cc:1434] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 16:17:28.366590 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-47801c6cdf6f461181a89246716390d7 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:28.367038 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-47801c6cdf6f461181a89246716390d7 on tablet 47801c6cdf6f461181a89246716390d7 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:28.376816 19073 ts_tablet_manager.cc:933] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:28.378899 19073 tablet_copy_client.cc:323] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:28.380514 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet b451ac456d424629ae5ce6040cb15bd2 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:28.380972 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet b451ac456d424629ae5ce6040cb15bd2 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-b451ac456d424629ae5ce6040cb15bd2
I20260317 16:17:28.388254 18222 tablet_copy_source_session.cc:215] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:28.391275 19073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b451ac456d424629ae5ce6040cb15bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:28.408721 19073 tablet_copy_client.cc:806] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:28.409349 19073 tablet_copy_client.cc:670] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:28.414911 19073 tablet_copy_client.cc:538] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:28.424314 19073 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:28.457113 19073 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:28.458406 19073 tablet_bootstrap.cc:492] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:28.458832 19081 ts_tablet_manager.cc:933] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:28.459033 19073 ts_tablet_manager.cc:1403] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.008s
I20260317 16:17:28.461979 19073 raft_consensus.cc:359] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.461970 19081 tablet_copy_client.cc:323] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:28.462880 19073 raft_consensus.cc:740] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:28.464147 19073 consensus_queue.cc:260] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.464867 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet e6aca2493a994ed888cd663ddcf97582 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:28.465453 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet e6aca2493a994ed888cd663ddcf97582 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-e6aca2493a994ed888cd663ddcf97582
I20260317 16:17:28.466683 19073 ts_tablet_manager.cc:1434] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 16:17:28.468756 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-b451ac456d424629ae5ce6040cb15bd2 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:28.469173 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-b451ac456d424629ae5ce6040cb15bd2 on tablet b451ac456d424629ae5ce6040cb15bd2 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:28.483278 18295 tablet_copy_source_session.cc:215] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:28.486856 19081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6aca2493a994ed888cd663ddcf97582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:28.506052 19081 tablet_copy_client.cc:806] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:28.506614 19081 tablet_copy_client.cc:670] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:28.510182 19081 tablet_copy_client.cc:538] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:28.520959 19081 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:28.542879 19081 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:28.543991 19081 tablet_bootstrap.cc:492] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:28.544720 19081 ts_tablet_manager.cc:1403] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.003s
I20260317 16:17:28.547575 19081 raft_consensus.cc:359] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.548182 19081 raft_consensus.cc:740] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:28.549505 19081 consensus_queue.cc:260] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:28.555612 19081 ts_tablet_manager.cc:1434] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260317 16:17:28.557214 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-e6aca2493a994ed888cd663ddcf97582 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:28.557613 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-e6aca2493a994ed888cd663ddcf97582 on tablet e6aca2493a994ed888cd663ddcf97582 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:28.622331 18816 raft_consensus.cc:1217] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:28.680160 18816 raft_consensus.cc:1217] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:28.935494 18816 raft_consensus.cc:1217] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:28.957219 18816 raft_consensus.cc:1217] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:29.122862 19001 raft_consensus.cc:1064] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:29.134169 19001 consensus_queue.cc:237] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.150713 18816 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.150997 18284 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.153185 18136 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.153059 19023 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:29.155323 19064 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:29.156807 19001 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:29.181691 19023 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.194134 18137 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.213478 18281 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.209625 18816 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.224419 18031 catalog_manager.cc:5671] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:29.300045 19084 raft_consensus.cc:1064] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:29.308316 19084 consensus_queue.cc:237] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.344707 18131 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.346951 19090 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:29.352845 18816 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.363158 18211 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.356496 19090 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260317 16:17:29.384725 18211 consensus_queue.cc:237] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.390210 19041 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:29.401896 19023 raft_consensus.cc:1064] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:29.404670 19023 consensus_queue.cc:237] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.405153 18815 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.413684 18134 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.415937 19001 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:29.420331 19001 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:29.449362 19094 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.467644 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 47801c6cdf6f461181a89246716390d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:29.471685 18136 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.480755 18286 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.484237 19094 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:29.489465 18817 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.471836 18815 raft_consensus.cc:2955] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.494967 18135 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.508416 18031 catalog_manager.cc:5671] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:29.510324 19064 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:29.512144 19041 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.519272 18816 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.531778 19090 raft_consensus.cc:1064] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:29.519644 19094 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:29.554459 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 47801c6cdf6f461181a89246716390d7 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:29.550490 19090 consensus_queue.cc:237] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.588564 19023 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.592712 18137 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.616294 18136 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.641363 19105 tablet_replica.cc:333] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:29.655623 18032 catalog_manager.cc:5671] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:29.691103 19105 raft_consensus.cc:2243] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:29.694022 19105 raft_consensus.cc:2272] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:29.701426 18816 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.706753 18282 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.710723 19056 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:29.718734 19105 ts_tablet_manager.cc:1916] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:29.727666 18285 consensus_queue.cc:237] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.737879 19105 ts_tablet_manager.cc:1929] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:29.738591 19105 log.cc:1199] T 47801c6cdf6f461181a89246716390d7 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/47801c6cdf6f461181a89246716390d7
I20260317 16:17:29.746934 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 47801c6cdf6f461181a89246716390d7 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:29.751304 18815 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.751570 18032 catalog_manager.cc:5671] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:29.760357 19090 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:29.760854 18210 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.770418 18134 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.771703 18136 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.773494 19110 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:29.774616 19084 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.776127 19090 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:29.776800 18815 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.782203 18136 raft_consensus.cc:2955] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.791314 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9844da16b03545c5a9e5534b29b4343c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:29.795653 18210 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:29.803242 18031 catalog_manager.cc:5671] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:29.818110 18817 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.824534 19090 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:29.829583 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 9844da16b03545c5a9e5534b29b4343c with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:29.839244 19115 tablet_replica.cc:333] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:29.839332 19041 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.841025 19115 raft_consensus.cc:2243] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:29.848871 19115 raft_consensus.cc:2272] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:29.856508 18210 consensus_queue.cc:237] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.858659 19115 ts_tablet_manager.cc:1916] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:29.856649 18136 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.866923 18285 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.870020 18816 raft_consensus.cc:1275] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.871935 19064 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:29.889580 19115 ts_tablet_manager.cc:1929] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:29.896080 19115 log.cc:1199] T 9844da16b03545c5a9e5534b29b4343c P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/9844da16b03545c5a9e5534b29b4343c
I20260317 16:17:29.880352 18032 catalog_manager.cc:5671] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:29.893875 19094 consensus_queue.cc:1048] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260317 16:17:29.900997 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 9844da16b03545c5a9e5534b29b4343c (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:29.886945 18816 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.920087 18211 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.923970 19023 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.940675 18280 consensus_queue.cc:237] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:29.940682 18285 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.948613 18207 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.952733 18816 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:29.954320 19090 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:29.960783 19110 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:29.958734 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet b451ac456d424629ae5ce6040cb15bd2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:29.969134 19107 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.975247 18210 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.977651 18815 raft_consensus.cc:2955] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.985626 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e6aca2493a994ed888cd663ddcf97582 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:29.986826 18817 raft_consensus.cc:2955] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:29.976644 18031 catalog_manager.cc:5671] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:29.999884 18035 catalog_manager.cc:5671] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:30.008512 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet b451ac456d424629ae5ce6040cb15bd2 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:30.011023 19117 tablet_replica.cc:333] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:30.012188 19117 raft_consensus.cc:2243] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:30.013057 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet e6aca2493a994ed888cd663ddcf97582 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:30.013859 19117 raft_consensus.cc:2272] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:30.023125 19117 ts_tablet_manager.cc:1916] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:30.041729 19117 ts_tablet_manager.cc:1929] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:30.042168 19117 log.cc:1199] T b451ac456d424629ae5ce6040cb15bd2 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/b451ac456d424629ae5ce6040cb15bd2
I20260317 16:17:30.044399 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet b451ac456d424629ae5ce6040cb15bd2 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:30.044672 19117 tablet_replica.cc:333] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:30.057845 19117 raft_consensus.cc:2243] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:30.058564 19117 raft_consensus.cc:2272] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:30.062672 19117 ts_tablet_manager.cc:1916] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:30.080003 19117 ts_tablet_manager.cc:1929] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:30.080489 19117 log.cc:1199] T e6aca2493a994ed888cd663ddcf97582 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/e6aca2493a994ed888cd663ddcf97582
I20260317 16:17:30.081844 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet e6aca2493a994ed888cd663ddcf97582 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:31.077965 18136 consensus_queue.cc:237] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:31.083961 18280 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:31.084331 18100 consensus_peers.cc:597] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac85b776838146d9904478a4ef0ae824. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:31.084532 18210 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:31.085884 19120 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:31.087697 19120 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:31.097154 19113 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.103680 18211 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.115310 18285 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.136986 18211 consensus_queue.cc:237] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:31.138399 18031 catalog_manager.cc:5671] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.157140 18136 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:31.158162 18285 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:31.158525 18175 consensus_peers.cc:597] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdb770e745e0447faf1334e5d343ace0. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:31.161101 19094 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:31.162468 19091 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:31.180879 19091 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.184986 18136 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.202950 18285 consensus_queue.cc:237] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:31.206939 18280 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.211227 18035 catalog_manager.cc:5671] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:31.220139 18249 consensus_peers.cc:597] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfaeab6c75f94e128924d831ee8ef55b. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:31.227995 18136 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:31.230425 18211 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:31.231333 19110 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:31.235520 19135 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:31.251998 19084 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.255544 18211 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.257364 18137 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.270157 18210 consensus_queue.cc:237] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:31.288236 18285 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:31.289572 18175 consensus_peers.cc:597] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87a9551ad4c24868805c90f4fa899ff0. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:31.289635 19118 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:31.295058 18137 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:31.294198 18031 catalog_manager.cc:5671] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.306602 19094 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:31.326448 19119 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.330185 18137 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.338392 18280 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.360163 18032 catalog_manager.cc:5671] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:31.783802 19144 ts_tablet_manager.cc:933] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:31.793830 19143 ts_tablet_manager.cc:933] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:31.789395 19145 ts_tablet_manager.cc:933] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:31.814754 19145 tablet_copy_client.cc:323] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:31.822145 19143 tablet_copy_client.cc:323] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:31.818320 19144 tablet_copy_client.cc:323] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
W20260317 16:17:31.826218 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 51654us
I20260317 16:17:31.827564 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet bfaeab6c75f94e128924d831ee8ef55b from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:31.828198 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet ac85b776838146d9904478a4ef0ae824 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:31.828424 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet bfaeab6c75f94e128924d831ee8ef55b from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-bfaeab6c75f94e128924d831ee8ef55b
I20260317 16:17:31.828894 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet ac85b776838146d9904478a4ef0ae824 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-ac85b776838146d9904478a4ef0ae824
I20260317 16:17:31.848073 18147 tablet_copy_source_session.cc:215] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:31.851675 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet fdb770e745e0447faf1334e5d343ace0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:31.852583 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet fdb770e745e0447faf1334e5d343ace0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-fdb770e745e0447faf1334e5d343ace0
I20260317 16:17:31.857487 18295 tablet_copy_source_session.cc:215] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:31.880213 19144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac85b776838146d9904478a4ef0ae824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:31.881821 19143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaeab6c75f94e128924d831ee8ef55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:31.897832 18222 tablet_copy_source_session.cc:215] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:31.905447 19145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb770e745e0447faf1334e5d343ace0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:31.905628 19152 ts_tablet_manager.cc:933] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:31.915277 19152 tablet_copy_client.cc:323] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:31.919358 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 87a9551ad4c24868805c90f4fa899ff0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:31.920120 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 87a9551ad4c24868805c90f4fa899ff0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-87a9551ad4c24868805c90f4fa899ff0
I20260317 16:17:31.929082 19143 tablet_copy_client.cc:806] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:31.930052 19143 tablet_copy_client.cc:670] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:31.941309 19143 tablet_copy_client.cc:538] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:31.947767 19144 tablet_copy_client.cc:806] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:31.948431 19144 tablet_copy_client.cc:670] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:31.955914 19144 tablet_copy_client.cc:538] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:31.966135 18222 tablet_copy_source_session.cc:215] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:31.970738 19143 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:31.971752 19152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a9551ad4c24868805c90f4fa899ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:31.984081 19144 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:31.984160 19145 tablet_copy_client.cc:806] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:31.984848 19145 tablet_copy_client.cc:670] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:31.993000 19145 tablet_copy_client.cc:538] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:32.015158 19145 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:32.056394 19152 tablet_copy_client.cc:806] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:32.057021 19152 tablet_copy_client.cc:670] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:32.080531 19144 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:32.081602 19144 tablet_bootstrap.cc:492] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:32.082345 19143 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:32.083503 19143 tablet_bootstrap.cc:492] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:32.083694 19144 ts_tablet_manager.cc:1403] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.100s user 0.038s sys 0.030s
I20260317 16:17:32.084271 19143 ts_tablet_manager.cc:1403] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.114s user 0.022s sys 0.008s
I20260317 16:17:32.084587 19145 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:32.085621 19145 tablet_bootstrap.cc:492] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:32.086380 19145 ts_tablet_manager.cc:1403] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.072s user 0.023s sys 0.010s
I20260317 16:17:32.087857 19143 raft_consensus.cc:359] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.087611 19144 raft_consensus.cc:359] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.089277 19143 raft_consensus.cc:740] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:32.089632 19144 raft_consensus.cc:740] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:32.090509 19144 consensus_queue.cc:260] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.090342 19143 consensus_queue.cc:260] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.093626 19145 raft_consensus.cc:359] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.094607 19145 raft_consensus.cc:740] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:32.096187 19152 tablet_copy_client.cc:538] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:32.101580 19144 ts_tablet_manager.cc:1434] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260317 16:17:32.105017 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-ac85b776838146d9904478a4ef0ae824 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:32.105676 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-ac85b776838146d9904478a4ef0ae824 on tablet ac85b776838146d9904478a4ef0ae824 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:32.109076 19152 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:32.109820 19145 consensus_queue.cc:260] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.121006 19145 ts_tablet_manager.cc:1434] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260317 16:17:32.142374 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-fdb770e745e0447faf1334e5d343ace0 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:32.143456 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-fdb770e745e0447faf1334e5d343ace0 on tablet fdb770e745e0447faf1334e5d343ace0 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:32.223564 18815 raft_consensus.cc:1217] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:32.270404 18812 raft_consensus.cc:1217] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:32.344660 19143 ts_tablet_manager.cc:1434] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.260s user 0.009s sys 0.011s
I20260317 16:17:32.348140 18815 raft_consensus.cc:1217] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:32.363515 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-bfaeab6c75f94e128924d831ee8ef55b received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:32.375926 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-bfaeab6c75f94e128924d831ee8ef55b on tablet bfaeab6c75f94e128924d831ee8ef55b with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:32.377205 19152 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:32.378825 19152 tablet_bootstrap.cc:492] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:32.379595 19152 ts_tablet_manager.cc:1403] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.271s user 0.025s sys 0.008s
I20260317 16:17:32.385123 19152 raft_consensus.cc:359] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:32.386204 19152 raft_consensus.cc:740] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:32.390319 19152 consensus_queue.cc:260] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:32.413096 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 45824us
I20260317 16:17:32.435338 18812 raft_consensus.cc:1217] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:32.441802 19152 ts_tablet_manager.cc:1434] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.062s user 0.008s sys 0.000s
I20260317 16:17:32.469797 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-87a9551ad4c24868805c90f4fa899ff0 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:32.470454 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-87a9551ad4c24868805c90f4fa899ff0 on tablet 87a9551ad4c24868805c90f4fa899ff0 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:32.553130 19170 raft_consensus.cc:1064] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:32.555913 19170 consensus_queue.cc:237] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:32.562780 18815 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.564949 19163 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:32.566393 18279 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.569849 19163 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:32.571528 18136 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.577481 19170 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:32.599689 19150 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.601230 18136 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.613847 18280 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.631667 18815 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.647959 18032 catalog_manager.cc:5671] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:32.679692 19135 raft_consensus.cc:1064] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:32.682013 19135 consensus_queue.cc:237] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:32.694172 18207 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.694661 18815 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.695083 18135 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.696112 19172 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:32.698009 19135 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:32.698856 19174 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:32.709878 19172 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.714545 18210 consensus_queue.cc:237] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:32.716733 18137 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.725170 18136 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:32.727736 18815 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:32.740424 19150 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:32.743363 19146 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:32.753544 18815 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.811980 19167 raft_consensus.cc:1064] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:32.808607 19163 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.819169 19164 raft_consensus.cc:1064] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:32.817564 18136 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.819996 18031 catalog_manager.cc:5671] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.821485 19167 consensus_queue.cc:237] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:32.827040 19164 consensus_queue.cc:237] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:32.883767 18280 consensus_queue.cc:237] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:32.885814 18279 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.886350 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet fdb770e745e0447faf1334e5d343ace0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:32.887831 18211 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.889142 19164 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:32.891433 18279 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.891417 19171 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:32.898198 19133 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:32.898669 18816 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:32.906214 18137 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:32.892789 18210 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.899729 18815 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.909732 19164 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:32.903546 18035 catalog_manager.cc:5671] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.911636 19139 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:32.924170 19139 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:32.932742 19172 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.941738 18812 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.944238 18815 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.939857 18137 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:32.962445 19147 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:32.969480 18817 raft_consensus.cc:2955] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.975340 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet fdb770e745e0447faf1334e5d343ace0 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:32.976969 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet bfaeab6c75f94e128924d831ee8ef55b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:32.976639 18135 raft_consensus.cc:2955] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:32.995376 19173 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.010058 19147 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260317 16:17:33.051198 18210 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
W20260317 16:17:33.065263 19192 log.cc:927] Time spent T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.106s user 0.003s sys 0.000s
I20260317 16:17:33.070922 18285 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.071378 18035 catalog_manager.cc:5671] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:33.085070 18812 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.118333 19194 tablet_replica.cc:333] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:33.120222 19194 raft_consensus.cc:2243] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:33.136574 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet bfaeab6c75f94e128924d831ee8ef55b with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:33.123339 18031 catalog_manager.cc:5671] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:33.138993 19162 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.151167 19194 raft_consensus.cc:2272] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:33.155205 19194 ts_tablet_manager.cc:1916] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:33.160892 18285 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.168481 19199 tablet_replica.cc:333] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:33.170877 19199 raft_consensus.cc:2243] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:33.171736 19199 raft_consensus.cc:2272] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:33.174295 18132 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.187391 19199 ts_tablet_manager.cc:1916] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:33.190867 18035 catalog_manager.cc:5671] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
W20260317 16:17:33.203167 19198 log.cc:927] Time spent T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260317 16:17:33.208599 18132 consensus_queue.cc:237] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:33.208660 19199 ts_tablet_manager.cc:1929] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:33.212679 19199 log.cc:1199] T bfaeab6c75f94e128924d831ee8ef55b P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/bfaeab6c75f94e128924d831ee8ef55b
I20260317 16:17:33.214636 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet bfaeab6c75f94e128924d831ee8ef55b (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:33.218452 18285 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:33.217099 18812 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.220026 18815 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:33.219900 19171 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:33.220952 19194 ts_tablet_manager.cc:1929] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:33.221418 19194 log.cc:1199] T fdb770e745e0447faf1334e5d343ace0 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/fdb770e745e0447faf1334e5d343ace0
I20260317 16:17:33.221702 19173 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:33.223403 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet fdb770e745e0447faf1334e5d343ace0 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:33.229439 19173 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.231657 18284 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.238754 18817 raft_consensus.cc:2955] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.259174 18207 consensus_queue.cc:237] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:33.261354 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet ac85b776838146d9904478a4ef0ae824 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:33.267287 18132 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:33.267601 18034 catalog_manager.cc:5671] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.269224 19170 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:33.274487 18812 raft_consensus.cc:1275] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:33.278024 19170 consensus_queue.cc:1048] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:33.282402 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet ac85b776838146d9904478a4ef0ae824 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:33.287217 19199 tablet_replica.cc:333] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:33.290729 19199 raft_consensus.cc:2243] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:33.296248 19160 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.306749 18132 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.295528 19199 raft_consensus.cc:2272] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:33.311138 18812 raft_consensus.cc:2955] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:33.314985 19199 ts_tablet_manager.cc:1916] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:33.316104 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 87a9551ad4c24868805c90f4fa899ff0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:33.340819 18035 catalog_manager.cc:5671] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:33.356434 19199 ts_tablet_manager.cc:1929] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:33.356982 19199 log.cc:1199] T ac85b776838146d9904478a4ef0ae824 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/ac85b776838146d9904478a4ef0ae824
I20260317 16:17:33.359331 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet ac85b776838146d9904478a4ef0ae824 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:33.359299 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a9551ad4c24868805c90f4fa899ff0 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:33.360574 19194 tablet_replica.cc:333] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:33.361577 19194 raft_consensus.cc:2243] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:33.362427 19194 raft_consensus.cc:2272] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:33.367455 19194 ts_tablet_manager.cc:1916] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:33.385007 19194 ts_tablet_manager.cc:1929] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:33.385378 19194 log.cc:1199] T 87a9551ad4c24868805c90f4fa899ff0 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/87a9551ad4c24868805c90f4fa899ff0
I20260317 16:17:33.386897 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 87a9551ad4c24868805c90f4fa899ff0 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:34.403939 18207 consensus_queue.cc:237] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:34.412156 18285 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.412472 18137 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:34.413025 18175 consensus_peers.cc:597] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c94382dceadb48228dd3467bf673a11a. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:34.414047 19197 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:34.414976 19160 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:34.422655 19170 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.424307 18285 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.424300 18137 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.440730 18210 consensus_queue.cc:237] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:34.461670 18035 catalog_manager.cc:5671] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.475790 18285 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.477865 19206 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260317 16:17:34.480302 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 44162.8us
I20260317 16:17:34.485395 18132 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.498960 19160 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:17:34.504714 18175 consensus_peers.cc:597] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b2a4d319ba9480b9c950e12b7329101. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:34.519629 19160 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.522459 18284 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.559569 18137 consensus_queue.cc:237] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:34.564888 18132 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.591835 18034 catalog_manager.cc:5671] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.605688 18210 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.606331 18285 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.607577 19201 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260317 16:17:34.611486 18100 consensus_peers.cc:597] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb31cad8b88240acb9c101027ea444c1. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:34.621865 19171 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:34.629824 18210 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.628968 19216 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.634301 18285 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.658360 18132 consensus_queue.cc:237] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:34.681643 18034 catalog_manager.cc:5671] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:34.686100 18100 consensus_peers.cc:597] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70f2fc6f0fd640fcb531a01d7881a614. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:34.686710 18211 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.687075 18285 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:34.688323 19201 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:34.690845 19216 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:34.703845 19216 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.707430 18211 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.715091 18285 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:34.737810 18035 catalog_manager.cc:5671] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:34.988986 19222 ts_tablet_manager.cc:933] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:34.993129 19222 tablet_copy_client.cc:323] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:34.996493 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet fb31cad8b88240acb9c101027ea444c1 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:34.997080 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet fb31cad8b88240acb9c101027ea444c1 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-fb31cad8b88240acb9c101027ea444c1
I20260317 16:17:35.005514 18147 tablet_copy_source_session.cc:215] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:35.008749 19224 ts_tablet_manager.cc:933] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:35.009289 19222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb31cad8b88240acb9c101027ea444c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:35.012925 19224 tablet_copy_client.cc:323] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:35.014830 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet c94382dceadb48228dd3467bf673a11a from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:35.015381 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet c94382dceadb48228dd3467bf673a11a from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-c94382dceadb48228dd3467bf673a11a
I20260317 16:17:35.021492 18222 tablet_copy_source_session.cc:215] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:35.028048 19222 tablet_copy_client.cc:806] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:35.028256 19224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c94382dceadb48228dd3467bf673a11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:35.028717 19222 tablet_copy_client.cc:670] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:35.032819 19222 tablet_copy_client.cc:538] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:35.044232 19222 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:35.048058 19224 tablet_copy_client.cc:806] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:35.048728 19224 tablet_copy_client.cc:670] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:35.052768 19224 tablet_copy_client.cc:538] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:35.075692 19224 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:35.086222 19222 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:35.087483 19222 tablet_bootstrap.cc:492] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:35.088250 19222 ts_tablet_manager.cc:1403] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.008s
I20260317 16:17:35.091447 19222 raft_consensus.cc:359] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.092352 19222 raft_consensus.cc:740] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:35.093154 19222 consensus_queue.cc:260] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.108465 19222 ts_tablet_manager.cc:1434] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.020s user 0.010s sys 0.005s
I20260317 16:17:35.111150 19228 ts_tablet_manager.cc:933] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:35.113957 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-fb31cad8b88240acb9c101027ea444c1 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:35.114449 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-fb31cad8b88240acb9c101027ea444c1 on tablet fb31cad8b88240acb9c101027ea444c1 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:35.129329 19228 tablet_copy_client.cc:323] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:35.132062 19224 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:35.133224 19224 tablet_bootstrap.cc:492] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:35.135142 19224 ts_tablet_manager.cc:1403] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.059s user 0.028s sys 0.005s
I20260317 16:17:35.137470 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 0b2a4d319ba9480b9c950e12b7329101 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:35.138078 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 0b2a4d319ba9480b9c950e12b7329101 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-0b2a4d319ba9480b9c950e12b7329101
I20260317 16:17:35.138901 19224 raft_consensus.cc:359] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.140228 19224 raft_consensus.cc:740] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:35.140982 19224 consensus_queue.cc:260] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.144119 19224 ts_tablet_manager.cc:1434] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260317 16:17:35.149592 18222 tablet_copy_source_session.cc:215] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:35.152585 19222 ts_tablet_manager.cc:933] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:35.155763 19222 tablet_copy_client.cc:323] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:35.158488 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 70f2fc6f0fd640fcb531a01d7881a614 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:35.159070 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 70f2fc6f0fd640fcb531a01d7881a614 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-70f2fc6f0fd640fcb531a01d7881a614
I20260317 16:17:35.163266 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-c94382dceadb48228dd3467bf673a11a received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:35.163714 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-c94382dceadb48228dd3467bf673a11a on tablet c94382dceadb48228dd3467bf673a11a with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:35.164389 19228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2a4d319ba9480b9c950e12b7329101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:35.166607 18147 tablet_copy_source_session.cc:215] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:35.170998 19222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f2fc6f0fd640fcb531a01d7881a614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:35.191597 19222 tablet_copy_client.cc:806] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:35.192240 19222 tablet_copy_client.cc:670] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:35.198398 19222 tablet_copy_client.cc:538] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:35.199877 19228 tablet_copy_client.cc:806] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:35.200536 19228 tablet_copy_client.cc:670] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:35.208806 19228 tablet_copy_client.cc:538] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:35.212604 19222 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:35.224170 19228 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:35.241981 19222 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:35.243152 19222 tablet_bootstrap.cc:492] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:35.243978 19222 ts_tablet_manager.cc:1403] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.009s
I20260317 16:17:35.247169 19222 raft_consensus.cc:359] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.248206 19222 raft_consensus.cc:740] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:35.249064 19222 consensus_queue.cc:260] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.252010 19222 ts_tablet_manager.cc:1434] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260317 16:17:35.258018 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-70f2fc6f0fd640fcb531a01d7881a614 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:35.258481 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-70f2fc6f0fd640fcb531a01d7881a614 on tablet 70f2fc6f0fd640fcb531a01d7881a614 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:35.271405 19228 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:35.272410 19228 tablet_bootstrap.cc:492] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:35.273090 19228 ts_tablet_manager.cc:1403] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.004s
I20260317 16:17:35.275988 19228 raft_consensus.cc:359] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.276805 19228 raft_consensus.cc:740] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:35.277498 19228 consensus_queue.cc:260] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:35.280213 19228 ts_tablet_manager.cc:1434] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260317 16:17:35.281692 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-0b2a4d319ba9480b9c950e12b7329101 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:35.282125 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-0b2a4d319ba9480b9c950e12b7329101 on tablet 0b2a4d319ba9480b9c950e12b7329101 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:35.424800 18812 raft_consensus.cc:1217] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:35.600234 18817 raft_consensus.cc:1217] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:35.606938 18817 raft_consensus.cc:1217] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:35.729175 18817 raft_consensus.cc:1217] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:36.008639 19235 raft_consensus.cc:1064] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:36.011189 19235 consensus_queue.cc:237] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.023204 18817 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.025381 19197 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:36.027171 18132 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.030562 18285 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.032083 19231 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:36.036195 19231 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:36.039340 19171 raft_consensus.cc:1064] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:36.046851 19171 consensus_queue.cc:237] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.060302 18812 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.060418 18207 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.062402 19216 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:36.070268 19216 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:36.082021 18285 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.086889 19201 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:36.090992 19197 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
W20260317 16:17:36.231491 18100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 85781.9us
I20260317 16:17:36.239820 18817 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.251346 19216 raft_consensus.cc:1064] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:36.267414 19160 raft_consensus.cc:1064] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:36.269760 19160 consensus_queue.cc:237] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.306890 19243 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.313540 18279 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.358975 19216 consensus_queue.cc:237] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
W20260317 16:17:36.360041 19241 log.cc:927] Time spent T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Append to log took a long time: real 0.223s user 0.003s sys 0.004s
I20260317 16:17:36.361155 18285 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260317 16:17:36.365542 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 50885.9us
I20260317 16:17:36.375703 18280 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.391399 19246 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
W20260317 16:17:36.393579 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 83021.7us
I20260317 16:17:36.399675 18034 catalog_manager.cc:5671] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.448403 18817 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.453786 18815 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.485750 18032 catalog_manager.cc:5671] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.504357 19262 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260317 16:17:36.543597 18132 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.549858 18817 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.551266 18211 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.552326 18206 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.554148 18131 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.561633 19269 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:36.562326 18281 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:36.566978 19248 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:36.577432 19171 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:36.596539 19262 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:36.650682 19267 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.676486 18211 consensus_queue.cc:237] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.653242 18207 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.698328 19253 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.722529 18282 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.724763 19260 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:36.725234 18817 raft_consensus.cc:1275] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.732542 18815 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.734509 19263 consensus_queue.cc:1048] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:36.734243 19197 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.751214 18281 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.734395 18282 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.743291 18817 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.743010 18279 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.748143 18135 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.761531 18132 consensus_queue.cc:237] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.768843 18812 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.771111 19271 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:36.771669 18032 catalog_manager.cc:5671] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:36.776777 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet c94382dceadb48228dd3467bf673a11a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:36.781034 18816 raft_consensus.cc:2955] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.790488 18035 catalog_manager.cc:5671] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.790189 19248 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.797863 18812 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.806182 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet fb31cad8b88240acb9c101027ea444c1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:36.810895 18281 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.817190 19267 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:36.824182 18136 consensus_queue.cc:237] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.824368 18031 catalog_manager.cc:5671] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.832487 18211 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.837596 18035 catalog_manager.cc:5671] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:36.842780 19267 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:36.843616 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet c94382dceadb48228dd3467bf673a11a with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:36.866389 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet fb31cad8b88240acb9c101027ea444c1 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:36.870205 18279 raft_consensus.cc:2955] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.884131 18812 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.885222 19271 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.887184 19248 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:36.913640 19281 tablet_replica.cc:333] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:36.915059 18206 consensus_queue.cc:237] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:36.910181 18817 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.920493 18211 raft_consensus.cc:2955] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.924587 18279 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.919899 19281 raft_consensus.cc:2243] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:36.940515 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 70f2fc6f0fd640fcb531a01d7881a614 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:36.942682 19281 raft_consensus.cc:2272] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:36.947125 19160 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:36.947775 18032 catalog_manager.cc:5671] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.955219 19280 tablet_replica.cc:333] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:36.956504 19280 raft_consensus.cc:2243] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:36.957619 19280 raft_consensus.cc:2272] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:36.958190 19197 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.963958 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 70f2fc6f0fd640fcb531a01d7881a614 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:36.967880 19280 ts_tablet_manager.cc:1916] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:36.968245 18282 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:36.973322 19281 ts_tablet_manager.cc:1916] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:36.974594 19282 tablet_replica.cc:333] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:36.978186 19282 raft_consensus.cc:2243] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:36.979094 19282 raft_consensus.cc:2272] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:36.979795 18812 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:36.989524 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 0b2a4d319ba9480b9c950e12b7329101 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:36.998950 19282 ts_tablet_manager.cc:1916] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:37.003327 19262 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:37.013437 18032 catalog_manager.cc:5671] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:37.016888 18817 raft_consensus.cc:2955] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:37.035102 19280 ts_tablet_manager.cc:1929] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:37.043054 19280 log.cc:1199] T c94382dceadb48228dd3467bf673a11a P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/c94382dceadb48228dd3467bf673a11a
I20260317 16:17:37.044862 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet c94382dceadb48228dd3467bf673a11a (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:37.053174 19281 ts_tablet_manager.cc:1929] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:37.053767 19281 log.cc:1199] T fb31cad8b88240acb9c101027ea444c1 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/fb31cad8b88240acb9c101027ea444c1
I20260317 16:17:37.055717 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet fb31cad8b88240acb9c101027ea444c1 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:37.046849 19282 ts_tablet_manager.cc:1929] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:37.059450 19282 log.cc:1199] T 70f2fc6f0fd640fcb531a01d7881a614 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/70f2fc6f0fd640fcb531a01d7881a614
I20260317 16:17:37.061194 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 70f2fc6f0fd640fcb531a01d7881a614 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:37.068382 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 0b2a4d319ba9480b9c950e12b7329101 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:37.069480 19280 tablet_replica.cc:333] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:37.070340 19280 raft_consensus.cc:2243] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:37.070950 19280 raft_consensus.cc:2272] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:37.073895 19280 ts_tablet_manager.cc:1916] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:37.092473 19280 ts_tablet_manager.cc:1929] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:37.092890 19280 log.cc:1199] T 0b2a4d319ba9480b9c950e12b7329101 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/0b2a4d319ba9480b9c950e12b7329101
I20260317 16:17:37.094281 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 0b2a4d319ba9480b9c950e12b7329101 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:37.474438 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:37.537386 18279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:37.537940 18279 raft_consensus.cc:606] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:37.540434 18279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:37.540951 18279 raft_consensus.cc:606] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:37.543771 18279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:37.544267 18279 raft_consensus.cc:606] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:37.546393 18279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:37.546934 18279 raft_consensus.cc:606] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:37.563987 18211 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:37.564559 18211 raft_consensus.cc:606] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:37.570372 18279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:37.571004 18279 raft_consensus.cc:606] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:37.572237 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260317 16:17:37.573431 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:37.574169 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:37.604054 19139 raft_consensus.cc:993] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.604537 19139 raft_consensus.cc:1081] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.605895 18135 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:37.606480 18135 raft_consensus.cc:493] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:37.606829 18135 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.612408 18135 raft_consensus.cc:515] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.613798 18135 leader_election.cc:290] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:37.614725 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:37.615285 18279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:37.615810 18211 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.616184 18279 raft_consensus.cc:3055] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:37.616499 18279 raft_consensus.cc:740] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:37.617230 18279 consensus_queue.cc:260] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.618350 18279 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.622107 18211 raft_consensus.cc:2468] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.623512 18102 leader_election.cc:304] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:37.624358 19267 raft_consensus.cc:2804] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:37.624837 19267 raft_consensus.cc:697] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:37.625227 18279 raft_consensus.cc:2468] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.626259 19267 consensus_queue.cc:237] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.654464 18035 catalog_manager.cc:5671] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:37.726613 19251 raft_consensus.cc:993] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:37.727314 19139 raft_consensus.cc:1081] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:37.729547 18817 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:37.730181 18817 raft_consensus.cc:493] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:37.730499 18817 raft_consensus.cc:3060] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.740325 18817 raft_consensus.cc:515] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:37.742954 18817 leader_election.cc:290] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:37.744056 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:37.744125 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "898420a8ccf84bbdba91710ca9c08ef1" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:37.744868 18282 raft_consensus.cc:3055] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:37.744834 18211 raft_consensus.cc:3060] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.745296 18282 raft_consensus.cc:740] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:37.746073 18282 consensus_queue.cc:260] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:37.747294 18282 raft_consensus.cc:3060] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.752332 18211 raft_consensus.cc:2468] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:37.753453 18781 leader_election.cc:304] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:37.753813 18282 raft_consensus.cc:2468] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:37.755648 19290 raft_consensus.cc:2804] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:37.757071 19290 raft_consensus.cc:697] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:37.757887 19251 raft_consensus.cc:993] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.758360 19139 raft_consensus.cc:1081] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.758067 19290 consensus_queue.cc:237] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:37.759831 18135 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:37.760284 18135 raft_consensus.cc:493] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:37.760643 18135 raft_consensus.cc:3060] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.766592 18135 raft_consensus.cc:515] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.768338 18135 leader_election.cc:290] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:37.767642 18035 catalog_manager.cc:5671] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:37.769075 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:37.769587 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ded1f55ede142cbbde7c675a0f72d15" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:37.769619 18211 raft_consensus.cc:3060] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.770136 18281 raft_consensus.cc:3055] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:37.770473 18281 raft_consensus.cc:740] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:37.771169 18281 consensus_queue.cc:260] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.772217 18281 raft_consensus.cc:3060] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.777259 18211 raft_consensus.cc:2468] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.778568 18102 leader_election.cc:304] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:37.779485 19267 raft_consensus.cc:2804] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:37.780191 19267 raft_consensus.cc:697] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:37.781093 19267 consensus_queue.cc:237] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.784461 18281 raft_consensus.cc:2468] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.797112 18035 catalog_manager.cc:5671] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:37.856624 19251 raft_consensus.cc:993] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.857440 19251 raft_consensus.cc:1081] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.859404 18135 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:37.859946 18135 raft_consensus.cc:493] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:37.860291 18135 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.869294 18135 raft_consensus.cc:515] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.871150 18135 leader_election.cc:290] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:37.872318 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:37.873158 18281 raft_consensus.cc:3055] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:37.872408 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:37.873484 18281 raft_consensus.cc:740] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:37.873714 18207 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.874230 18281 consensus_queue.cc:260] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.875867 18281 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.878973 18207 raft_consensus.cc:2468] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.879942 18102 leader_election.cc:304] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:37.881793 18281 raft_consensus.cc:2468] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.882550 19267 raft_consensus.cc:2804] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:37.883074 19267 raft_consensus.cc:697] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:37.883940 19267 consensus_queue.cc:237] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.899708 18035 catalog_manager.cc:5671] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:37.949982 19251 raft_consensus.cc:993] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.950697 19139 raft_consensus.cc:1081] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:37.953101 18135 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:37.956943 18135 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:37.957401 18135 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.962613 18135 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.964136 18135 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:37.965531 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:37.966138 18211 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.966536 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:37.967176 18281 raft_consensus.cc:3055] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:37.968343 18281 raft_consensus.cc:740] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:37.970484 18281 consensus_queue.cc:260] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.971809 18281 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:37.974593 18211 raft_consensus.cc:2468] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.976002 18102 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:37.976950 19271 raft_consensus.cc:2804] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:37.977423 19271 raft_consensus.cc:697] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:37.977789 18281 raft_consensus.cc:2468] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:37.978327 19271 consensus_queue.cc:237] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:37.991192 18035 catalog_manager.cc:5671] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:38.059226 19262 raft_consensus.cc:993] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:38.059719 19249 raft_consensus.cc:1081] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:38.062103 18282 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:38.065438 18282 raft_consensus.cc:493] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:38.066119 18282 raft_consensus.cc:3060] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.074443 18282 raft_consensus.cc:515] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.077620 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:38.078207 18135 raft_consensus.cc:3060] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.078491 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9545c351b28449c2a44da8b9a94c7f11" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:38.079279 18211 raft_consensus.cc:3055] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:38.079591 18211 raft_consensus.cc:740] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:38.080303 18211 consensus_queue.cc:260] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.081447 18211 raft_consensus.cc:3060] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.085589 18135 raft_consensus.cc:2468] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:38.086982 18249 leader_election.cc:304] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:38.088215 19251 raft_consensus.cc:2804] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:38.089145 19251 raft_consensus.cc:697] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:38.090358 19251 consensus_queue.cc:237] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.092114 18135 consensus_queue.cc:237] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:38.101392 18211 raft_consensus.cc:2468] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:38.102711 18282 leader_election.cc:290] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:38.120721 18281 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:38.121343 18100 consensus_peers.cc:597] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d88548519a0f473680897b8a7fc1e9b6. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:38.122726 19267 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:38.125063 18211 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:38.127038 19267 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.127501 18035 catalog_manager.cc:5671] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:38.138880 19267 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:38.143838 18211 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:38.140549 18281 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:38.167145 18083 auto_rebalancer.cc:275] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20260317 16:17:38.172369 18035 catalog_manager.cc:5671] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:38.186357 18083 auto_rebalancer.cc:798] Removing replace marker failed: Leader has not yet committed an operation in its own term
W20260317 16:17:38.186861 18083 auto_rebalancer.cc:806] Could not move replica: Incomplete: tablet 77f02a05d0ee4e73bafe3055cfcf02f2, TS 88b3a81306894b71bf2eacc88c770eaa -> TS 381cc8622e1f4a53a7ff2b23ed2bcd38 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:38.189182 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Incomplete: tablet 77f02a05d0ee4e73bafe3055cfcf02f2, TS 88b3a81306894b71bf2eacc88c770eaa -> TS 381cc8622e1f4a53a7ff2b23ed2bcd38 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.196596 18282 raft_consensus.cc:1275] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.198299 19271 consensus_queue.cc:1048] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:38.207057 18282 consensus_queue.cc:237] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.214870 18211 raft_consensus.cc:1275] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.216642 19271 consensus_queue.cc:1048] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.229672 18207 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:38.231069 19251 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.231489 18135 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:38.233096 19251 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.256414 19251 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
I20260317 16:17:38.257834 18207 raft_consensus.cc:1275] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.262702 19267 consensus_queue.cc:1048] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:38.271636 18136 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
I20260317 16:17:38.264487 18207 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
I20260317 16:17:38.336659 18031 catalog_manager.cc:5671] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
W20260317 16:17:38.350705 18083 auto_rebalancer.cc:806] Could not move replica: Incomplete: tablet a8ed2db954354e669de7909d315142aa, TS 94edf207909c4f8990bfe29ed1a87a6d -> TS 381cc8622e1f4a53a7ff2b23ed2bcd38 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:38.351145 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Incomplete: tablet a8ed2db954354e669de7909d315142aa, TS 94edf207909c4f8990bfe29ed1a87a6d -> TS 381cc8622e1f4a53a7ff2b23ed2bcd38 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.351938 18282 raft_consensus.cc:1275] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.354501 18282 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.356784 19267 consensus_queue.cc:1048] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:38.361203 19267 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:17:38.395057 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 87002.2us
I20260317 16:17:38.441489 18208 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:38.456396 19314 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260317 16:17:38.491156 18211 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.492504 19271 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.502154 18136 raft_consensus.cc:1275] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.503721 18285 raft_consensus.cc:1275] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:38.506301 19139 consensus_queue.cc:1048] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.509927 18206 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.506843 18281 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.506567 18280 consensus_queue.cc:237] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.511651 19271 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.528092 19314 consensus_queue.cc:1048] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:38.545425 18132 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:38.548753 19295 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:38.569701 19309 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:38.576443 18206 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:38.580550 19309 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260317 16:17:38.606017 18210 raft_consensus.cc:1275] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:38.613932 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:38.629997 19334 ts_tablet_manager.cc:933] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:38.634156 19317 consensus_queue.cc:1048] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:38.643217 19334 tablet_copy_client.cc:323] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:38.689092 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet d88548519a0f473680897b8a7fc1e9b6 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:38.735957 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet d88548519a0f473680897b8a7fc1e9b6 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-d88548519a0f473680897b8a7fc1e9b6
I20260317 16:17:38.746021 18206 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:38.746608 18206 raft_consensus.cc:606] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:38.757210 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47801c6cdf6f461181a89246716390d7"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:38.758309 18208 raft_consensus.cc:606] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:38.744774 19317 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
I20260317 16:17:38.775205 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:38.777158 18208 raft_consensus.cc:606] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:38.747009 18132 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
I20260317 16:17:38.806210 18147 tablet_copy_source_session.cc:215] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:38.825848 19334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88548519a0f473680897b8a7fc1e9b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:17:38.829993 18083 auto_rebalancer.cc:806] Could not move replica: Incomplete: tablet 94f2b31d108e461d932c2477e0f78fd0, TS 94edf207909c4f8990bfe29ed1a87a6d -> TS 381cc8622e1f4a53a7ff2b23ed2bcd38 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:38.830420 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Incomplete: tablet 94f2b31d108e461d932c2477e0f78fd0, TS 94edf207909c4f8990bfe29ed1a87a6d -> TS 381cc8622e1f4a53a7ff2b23ed2bcd38 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.832260 19249 raft_consensus.cc:993] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:38.832823 19249 raft_consensus.cc:1081] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:38.844099 19262 raft_consensus.cc:993] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:38.844564 19262 raft_consensus.cc:1081] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:38.849180 18281 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:38.849728 18281 raft_consensus.cc:493] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:38.849337 18032 catalog_manager.cc:5671] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:38.850101 18281 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.865226 18281 raft_consensus.cc:515] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
W20260317 16:17:38.869626 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 57498.1us
I20260317 16:17:38.870726 18281 leader_election.cc:290] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:38.872535 18132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:38.874388 18132 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.878223 18817 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47801c6cdf6f461181a89246716390d7"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57316
I20260317 16:17:38.878830 18817 raft_consensus.cc:493] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:38.879195 18817 raft_consensus.cc:3060] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.885344 18132 raft_consensus.cc:2468] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:38.888068 18249 leader_election.cc:304] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:38.889007 19317 raft_consensus.cc:2804] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:38.889492 19317 raft_consensus.cc:697] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:38.890595 19317 consensus_queue.cc:237] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.891273 18817 raft_consensus.cc:515] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:38.893714 18136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:38.894304 18136 raft_consensus.cc:606] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:38.894165 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:38.895754 18135 raft_consensus.cc:3060] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.899295 18817 leader_election.cc:290] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:38.901714 18210 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8284701fdaf640daa689a2534914514c"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:38.902446 18210 raft_consensus.cc:606] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:38.905045 18135 raft_consensus.cc:2468] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:38.905758 18281 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:38.906036 18779 leader_election.cc:304] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:38.906903 19314 raft_consensus.cc:2804] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:38.907433 19314 raft_consensus.cc:697] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:38.908347 19314 consensus_queue.cc:237] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:38.911075 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:38.911633 18206 raft_consensus.cc:3055] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:38.911953 18206 raft_consensus.cc:740] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:38.912868 18206 consensus_queue.cc:260] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:38.914381 18206 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.921195 19334 tablet_copy_client.cc:806] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:38.921844 19334 tablet_copy_client.cc:670] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:38.928486 19267 raft_consensus.cc:993] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:38.929463 19267 raft_consensus.cc:1081] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:38.931980 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57886
I20260317 16:17:38.932516 18812 raft_consensus.cc:493] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:38.932869 18812 raft_consensus.cc:3060] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.933364 19334 tablet_copy_client.cc:538] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:38.933249 18210 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } }
I20260317 16:17:38.929299 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47801c6cdf6f461181a89246716390d7" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:38.947162 18208 raft_consensus.cc:3055] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:38.949301 18135 raft_consensus.cc:1240] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Rejecting Update request from peer 88b3a81306894b71bf2eacc88c770eaa for earlier term 1. Current term is 2. Ops: []
I20260317 16:17:38.950872 18208 raft_consensus.cc:740] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:38.951421 18812 raft_consensus.cc:515] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:38.951915 19341 consensus_queue.cc:1059] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:38.954952 18812 leader_election.cc:290] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:38.955457 19334 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:38.956879 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:38.970243 18137 raft_consensus.cc:3055] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:38.982339 18137 raft_consensus.cc:740] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:38.983302 18137 consensus_queue.cc:260] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:38.987602 18137 raft_consensus.cc:3060] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.964991 18032 catalog_manager.cc:5671] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:38.959679 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f2fc6f0fd640fcb531a01d7881a614" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:38.978214 18208 consensus_queue.cc:260] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:38.906478 18281 raft_consensus.cc:606] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:38.993985 18211 raft_consensus.cc:3060] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.995018 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260317 16:17:38.995396 18208 raft_consensus.cc:3060] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:38.996876 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:38.997711 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:39.002298 18206 raft_consensus.cc:2468] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:39.004592 19334 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:39.005620 19334 tablet_bootstrap.cc:492] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:39.006844 19334 ts_tablet_manager.cc:1403] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.010s
I20260317 16:17:39.009887 19334 raft_consensus.cc:359] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:39.010845 19334 raft_consensus.cc:740] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:39.011569 19334 consensus_queue.cc:260] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:39.014308 18211 raft_consensus.cc:2468] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:39.017735 19334 ts_tablet_manager.cc:1434] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260317 16:17:39.020859 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-d88548519a0f473680897b8a7fc1e9b6 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:39.021582 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-d88548519a0f473680897b8a7fc1e9b6 on tablet d88548519a0f473680897b8a7fc1e9b6 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:39.022701 18781 leader_election.cc:304] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:39.023746 19330 raft_consensus.cc:2804] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:39.024364 19330 raft_consensus.cc:697] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:39.025621 19330 consensus_queue.cc:237] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.040958 18208 raft_consensus.cc:2468] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
W20260317 16:17:39.046639 19338 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260317 16:17:39.045754 18137 raft_consensus.cc:2468] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:39.049875 19341 raft_consensus.cc:993] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:39.050917 19341 raft_consensus.cc:1081] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:39.050745 18032 catalog_manager.cc:5671] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:39.053354 18280 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:39.053884 18280 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:39.054191 18280 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:39.057855 19337 raft_consensus.cc:993] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:39.058349 19337 raft_consensus.cc:1081] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:39.059937 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8284701fdaf640daa689a2534914514c"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57316
I20260317 16:17:39.060420 18812 raft_consensus.cc:493] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:39.060777 18812 raft_consensus.cc:3060] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:39.065901 18280 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:39.080120 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:39.080718 18136 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:39.089738 18136 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 3.
I20260317 16:17:39.080240 18280 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:39.090968 18249 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:39.092012 19251 raft_consensus.cc:2804] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:39.092509 19251 raft_consensus.cc:697] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:39.093444 19251 consensus_queue.cc:237] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:39.093348 18032 catalog_manager.cc:5671] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:39.077544 18812 raft_consensus.cc:515] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.121085 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8284701fdaf640daa689a2534914514c" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:39.121713 18136 raft_consensus.cc:3060] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:39.129232 18136 raft_consensus.cc:2468] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:39.130597 18779 leader_election.cc:304] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:39.131649 19330 raft_consensus.cc:2804] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:39.132117 19330 raft_consensus.cc:697] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:39.132987 19330 consensus_queue.cc:237] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.145437 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8284701fdaf640daa689a2534914514c" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:39.146173 18208 raft_consensus.cc:3055] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:39.150022 18817 raft_consensus.cc:1217] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:39.155020 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:39.156105 18206 raft_consensus.cc:3055] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:39.146525 18208 raft_consensus.cc:740] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:39.156625 18206 raft_consensus.cc:740] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:39.157665 18206 consensus_queue.cc:260] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:39.171689 18206 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:39.176179 18031 catalog_manager.cc:5671] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 2 to 3, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:39.178892 18206 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 3.
I20260317 16:17:39.178701 18035 catalog_manager.cc:5671] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:39.157698 18208 consensus_queue.cc:260] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.163256 19317 raft_consensus.cc:993] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:39.184350 18208 raft_consensus.cc:3060] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:39.184791 19317 raft_consensus.cc:1081] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:39.147810 18812 leader_election.cc:290] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:39.191424 18208 raft_consensus.cc:2468] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:39.193049 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:39.193645 18812 raft_consensus.cc:493] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:39.193984 18812 raft_consensus.cc:3060] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:39.207047 18812 raft_consensus.cc:515] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.208806 18812 leader_election.cc:290] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:39.210570 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:39.211172 18135 raft_consensus.cc:3060] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:39.218572 18135 raft_consensus.cc:2468] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:39.219568 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9844da16b03545c5a9e5534b29b4343c" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:39.220526 18285 raft_consensus.cc:3055] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:39.220908 18285 raft_consensus.cc:740] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:39.221601 18285 consensus_queue.cc:260] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.222966 18285 raft_consensus.cc:3060] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:39.223055 18779 leader_election.cc:304] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:39.225025 19330 raft_consensus.cc:2804] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:39.225517 19330 raft_consensus.cc:697] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:39.226348 19330 consensus_queue.cc:237] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.292212 18137 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:39.293686 18285 raft_consensus.cc:2468] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:39.294462 19317 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260317 16:17:39.299734 18211 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:39.302454 19317 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:39.309331 18031 catalog_manager.cc:5671] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:39.414034 18135 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.416905 19314 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:39.419440 18211 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.421133 19314 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:39.440829 18206 raft_consensus.cc:1275] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.443097 19314 consensus_queue.cc:1048] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:39.464545 18132 raft_consensus.cc:1275] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.468729 19330 consensus_queue.cc:1048] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:39.486892 18132 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:17:39.489475 19316 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:39.554320 18208 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:17:39.568055 19309 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:39.668395 18137 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.670085 19330 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:39.677630 18208 raft_consensus.cc:1275] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.681316 19330 consensus_queue.cc:1048] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:39.691496 19267 raft_consensus.cc:1064] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:39.694145 19267 consensus_queue.cc:237] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.719164 18285 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:39.722214 18137 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.725977 19327 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:39.729465 18208 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:39.731041 19267 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:39.722466 18812 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:39.733477 19330 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:39.733906 19295 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:39.758564 18282 raft_consensus.cc:1275] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:39.762411 19327 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.765285 19330 consensus_queue.cc:1048] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:39.771063 18282 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.785786 18812 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.853284 18031 catalog_manager.cc:5671] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.858421 18208 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.916137 18137 consensus_queue.cc:237] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:39.921742 18208 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:39.924232 19327 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:39.924484 18817 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:39.930181 19327 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:39.932281 19295 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.937261 18206 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.939543 18817 raft_consensus.cc:2955] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:39.952731 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet d88548519a0f473680897b8a7fc1e9b6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:39.960929 18035 catalog_manager.cc:5671] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:39.977869 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet d88548519a0f473680897b8a7fc1e9b6 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:39.980892 19373 tablet_replica.cc:333] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:39.982288 19373 raft_consensus.cc:2243] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:39.983594 19373 raft_consensus.cc:2272] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:39.987550 19373 ts_tablet_manager.cc:1916] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:40.007475 19373 ts_tablet_manager.cc:1929] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:40.007961 19373 log.cc:1199] T d88548519a0f473680897b8a7fc1e9b6 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/d88548519a0f473680897b8a7fc1e9b6
I20260317 16:17:40.009478 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet d88548519a0f473680897b8a7fc1e9b6 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:40.158355 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:40.208979 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "09f47966c253483393878fddf97ac206"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:40.209529 18208 raft_consensus.cc:606] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:40.212036 18285 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:40.212641 18285 raft_consensus.cc:606] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:40.215364 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:40.216012 18208 raft_consensus.cc:606] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:40.222831 18285 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:40.223946 18285 raft_consensus.cc:606] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:40.227643 18136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:40.228232 19317 raft_consensus.cc:993] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:40.228438 18136 raft_consensus.cc:606] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:40.228933 19317 raft_consensus.cc:1081] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:40.230481 18136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:40.231148 18136 raft_consensus.cc:493] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:40.231539 18136 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:40.231951 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:40.232528 18208 raft_consensus.cc:606] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:40.233579 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260317 16:17:40.235337 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:40.235982 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:40.237244 18136 raft_consensus.cc:515] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.239948 18136 leader_election.cc:290] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 3 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:40.240684 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:40.241385 18208 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:40.244186 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee9125e28d14f3c8f458dda86c4ee15" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:40.244807 18285 raft_consensus.cc:3055] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:40.248112 18285 raft_consensus.cc:740] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:40.251463 18285 consensus_queue.cc:260] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.253196 18285 raft_consensus.cc:3060] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:40.254277 18208 raft_consensus.cc:2468] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 3.
I20260317 16:17:40.256735 18102 leader_election.cc:304] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:40.257503 19321 raft_consensus.cc:2804] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:40.258002 19321 raft_consensus.cc:697] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:40.258858 19321 consensus_queue.cc:237] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.263527 18285 raft_consensus.cc:2468] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 3.
I20260317 16:17:40.271198 18031 catalog_manager.cc:5671] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 2 to 3, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 3 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:40.281240 19321 raft_consensus.cc:993] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:40.282281 19295 raft_consensus.cc:1081] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:40.284294 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57886
I20260317 16:17:40.284988 18812 raft_consensus.cc:493] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:40.285337 18812 raft_consensus.cc:3060] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.295111 18812 raft_consensus.cc:515] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.303141 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:40.304283 18137 raft_consensus.cc:3055] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:40.304587 18137 raft_consensus.cc:740] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:40.305294 18137 consensus_queue.cc:260] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.306346 18137 raft_consensus.cc:3060] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.306668 18812 leader_election.cc:290] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:40.308027 18285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac85b776838146d9904478a4ef0ae824" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:40.308601 18285 raft_consensus.cc:3060] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.316355 18285 raft_consensus.cc:2468] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:40.318886 18137 raft_consensus.cc:2468] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:40.318593 18782 leader_election.cc:304] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:40.320116 19330 raft_consensus.cc:2804] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:40.320873 19330 raft_consensus.cc:697] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:40.322299 19330 consensus_queue.cc:237] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.332012 18031 catalog_manager.cc:5671] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:40.390128 19317 raft_consensus.cc:993] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:40.390671 19317 raft_consensus.cc:1081] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:40.392381 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:40.392910 18812 raft_consensus.cc:493] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:40.393229 19337 raft_consensus.cc:993] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:40.393257 18812 raft_consensus.cc:3060] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.393855 19337 raft_consensus.cc:1081] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:40.395687 18285 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:40.396173 18285 raft_consensus.cc:493] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:40.396461 18285 raft_consensus.cc:3060] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.404054 18812 raft_consensus.cc:515] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.406301 18812 leader_election.cc:290] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:40.407114 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:40.407436 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e145759d6de4334931c0039a97abdd8" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:40.407788 18137 raft_consensus.cc:3060] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.407670 18285 raft_consensus.cc:515] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.408021 18281 raft_consensus.cc:3055] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:40.408455 18281 raft_consensus.cc:740] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:40.409150 18281 consensus_queue.cc:260] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.410122 18281 raft_consensus.cc:3060] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.410390 18285 leader_election.cc:290] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:40.411031 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:40.411551 18208 raft_consensus.cc:3055] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:40.411842 18208 raft_consensus.cc:740] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:40.412276 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "702896c0cf9948d68e4c4148743bc5c8" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:40.412789 18136 raft_consensus.cc:3060] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.412537 18208 consensus_queue.cc:260] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.413717 18208 raft_consensus.cc:3060] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.415076 18137 raft_consensus.cc:2468] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:40.418546 18281 raft_consensus.cc:2468] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:40.419610 18779 leader_election.cc:304] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:40.420156 18136 raft_consensus.cc:2468] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:40.420918 19330 raft_consensus.cc:2804] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:40.421223 18249 leader_election.cc:304] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:40.421403 19330 raft_consensus.cc:697] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:40.421484 18208 raft_consensus.cc:2468] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:40.422094 19251 raft_consensus.cc:2804] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:40.422549 19251 raft_consensus.cc:697] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:40.423275 19251 consensus_queue.cc:237] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.422571 19330 consensus_queue.cc:237] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.446249 18031 catalog_manager.cc:5671] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:40.448362 18035 catalog_manager.cc:5671] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:40.590468 19262 raft_consensus.cc:993] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:40.591535 19337 raft_consensus.cc:1081] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:40.594130 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "09f47966c253483393878fddf97ac206"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57316
I20260317 16:17:40.595985 18812 raft_consensus.cc:493] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:40.596386 18812 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:40.604763 18812 raft_consensus.cc:515] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.606107 18812 leader_election.cc:290] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:40.607060 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:40.607326 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09f47966c253483393878fddf97ac206" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:40.607710 18136 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:40.607833 18206 raft_consensus.cc:3055] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:40.608182 18206 raft_consensus.cc:740] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:40.609113 18206 consensus_queue.cc:260] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.610988 18206 raft_consensus.cc:3060] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:40.617486 18136 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 3.
I20260317 16:17:40.619455 18779 leader_election.cc:304] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:40.620535 19330 raft_consensus.cc:2804] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:40.620786 18206 raft_consensus.cc:2468] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 3.
I20260317 16:17:40.621078 19330 raft_consensus.cc:697] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:40.622347 19330 consensus_queue.cc:237] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:40.632118 19337 raft_consensus.cc:993] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:40.632786 19337 raft_consensus.cc:1081] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:40.635200 18281 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:40.635851 18281 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:40.636232 18281 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.647159 18281 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.649085 18281 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:40.649962 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:40.650043 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:40.650717 18206 raft_consensus.cc:3055] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:40.651463 18137 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.651456 18206 raft_consensus.cc:740] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:40.651955 18035 catalog_manager.cc:5671] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 2 to 3, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 3 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:40.653255 18206 consensus_queue.cc:260] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.654448 18206 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:40.659339 18137 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:40.664289 18249 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:40.666134 18206 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:40.666730 19251 raft_consensus.cc:2804] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:40.667867 19251 raft_consensus.cc:697] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:40.669502 19251 consensus_queue.cc:237] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:40.691241 18206 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:40.692751 19321 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:40.693100 18035 catalog_manager.cc:5671] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:40.710464 18281 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:40.713632 19267 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:40.904733 18137 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:40.906023 19330 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:40.916044 18285 raft_consensus.cc:1275] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:40.917779 19387 consensus_queue.cc:1048] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:40.972421 18206 raft_consensus.cc:1275] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:40.973618 19251 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:40.988693 18137 raft_consensus.cc:1275] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:40.990162 19251 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:41.008340 18136 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:41.009838 19387 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:41.012490 18285 raft_consensus.cc:1275] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:41.014015 19387 consensus_queue.cc:1048] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:41.076336 18136 consensus_queue.cc:237] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:41.084014 18100 consensus_peers.cc:597] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07a6f1b891ac4c3492ae2548ff2fc357. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:41.084055 18285 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:41.085124 18208 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:41.085938 19321 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:41.086817 19295 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:41.091924 18136 raft_consensus.cc:1275] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:41.093864 19251 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:41.120355 19295 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.130311 18285 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.131258 18208 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.146724 18206 raft_consensus.cc:1275] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:41.153990 19309 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:41.155043 18032 catalog_manager.cc:5671] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.164983 18285 consensus_queue.cc:237] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:41.176509 18249 consensus_peers.cc:597] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8ed2db954354e669de7909d315142aa. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:41.177796 18208 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:41.178444 18137 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:41.180330 19251 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:41.181768 19317 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:41.189721 19251 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.191108 18137 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.194341 18208 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.197938 18206 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:41.200450 19328 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:41.207350 18285 consensus_queue.cc:237] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:41.212950 18032 catalog_manager.cc:5671] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:41.216724 18211 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:41.218170 19251 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:41.219511 18136 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:41.219980 18249 consensus_peers.cc:597] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7a4647163b461f87a0a249491e566c. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:41.220927 19251 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:41.260341 18137 raft_consensus.cc:1275] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:41.262750 19330 consensus_queue.cc:1048] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:41.275535 19317 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.276841 18208 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.303514 18135 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.308158 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:41.308085 18032 catalog_manager.cc:5671] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.311173 18208 consensus_queue.cc:237] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:41.330917 18175 consensus_peers.cc:597] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e390c786df0b48878244a08e889950a3. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:41.333534 18285 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:41.335075 19262 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:41.336822 18136 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:41.338387 19393 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:41.359583 19337 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.360919 18135 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.370740 18285 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.389748 18032 catalog_manager.cc:5671] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:41.397275 18282 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:41.398058 18282 raft_consensus.cc:606] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:41.414242 18211 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "519bc447dbc1455c844945ba63c96914"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:41.414896 18211 raft_consensus.cc:606] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:41.419811 18211 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:41.420367 18211 raft_consensus.cc:606] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:41.423810 18285 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:41.424921 18285 raft_consensus.cc:606] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:41.429558 18135 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:41.432399 18135 raft_consensus.cc:606] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:41.435078 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260317 16:17:41.436728 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:41.437446 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:41.498159 19321 raft_consensus.cc:993] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:41.498657 19321 raft_consensus.cc:1081] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:41.500286 18211 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:41.500938 18211 raft_consensus.cc:493] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:41.501287 18211 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:41.510848 18211 raft_consensus.cc:515] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.516136 18211 leader_election.cc:290] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:41.516645 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:41.517481 18281 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:41.523838 18281 raft_consensus.cc:2468] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:41.525553 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdf3aa784b554e5386dcc11e2550e2f5" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:41.525627 18178 leader_election.cc:304] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:41.526404 18137 raft_consensus.cc:3055] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:41.526751 18137 raft_consensus.cc:740] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:41.526751 19337 raft_consensus.cc:2804] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:41.527415 19337 raft_consensus.cc:697] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:41.527709 19262 raft_consensus.cc:993] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:41.527566 18137 consensus_queue.cc:260] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.528409 19337 consensus_queue.cc:237] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.529088 18137 raft_consensus.cc:3060] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:41.529290 19393 raft_consensus.cc:1081] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:41.531075 18281 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:41.531647 18281 raft_consensus.cc:493] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:41.532114 18281 raft_consensus.cc:3060] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.544299 18281 raft_consensus.cc:515] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.551158 19262 raft_consensus.cc:993] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:41.551720 19393 raft_consensus.cc:1081] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:41.553284 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:41.553408 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aafcd49c9d54f548b25a51935f79523" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:41.553887 18135 raft_consensus.cc:3060] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.554081 18208 raft_consensus.cc:3055] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:41.554467 18208 raft_consensus.cc:740] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:41.555166 18208 consensus_queue.cc:260] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.556032 18208 raft_consensus.cc:3060] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.557201 18282 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "519bc447dbc1455c844945ba63c96914"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:41.557727 18282 raft_consensus.cc:493] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:41.558032 18282 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.560392 18137 raft_consensus.cc:2468] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:41.560714 18135 raft_consensus.cc:2468] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:41.553251 18281 leader_election.cc:290] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:41.563400 18249 leader_election.cc:304] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:41.567121 19309 raft_consensus.cc:2804] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:41.567633 19309 raft_consensus.cc:697] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:41.568478 19309 consensus_queue.cc:237] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.571861 18208 raft_consensus.cc:2468] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:41.576283 19251 raft_consensus.cc:993] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:41.576908 19251 raft_consensus.cc:1081] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:41.578444 18137 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:41.578974 18137 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:41.579306 18137 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:41.582325 19414 ts_tablet_manager.cc:933] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:41.587097 18282 raft_consensus.cc:515] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.589999 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:41.590567 18211 raft_consensus.cc:3055] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:41.590891 18211 raft_consensus.cc:740] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:41.591514 18211 consensus_queue.cc:260] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.592690 18211 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.599241 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:41.601995 19417 ts_tablet_manager.cc:933] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:41.603811 18032 catalog_manager.cc:5671] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 2 to 3, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:41.602227 18136 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.605803 18211 raft_consensus.cc:2468] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:41.607540 18251 leader_election.cc:304] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:41.608405 19317 raft_consensus.cc:2804] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:41.608898 19317 raft_consensus.cc:697] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:41.609786 19317 consensus_queue.cc:237] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.610270 19414 tablet_copy_client.cc:323] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:41.612141 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet a8ed2db954354e669de7909d315142aa from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:41.612704 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet a8ed2db954354e669de7909d315142aa from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-a8ed2db954354e669de7909d315142aa
I20260317 16:17:41.606225 18282 leader_election.cc:290] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:41.623507 18295 tablet_copy_source_session.cc:215] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:41.603184 18137 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.629431 19414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ed2db954354e669de7909d315142aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:41.630034 18137 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 4 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:41.630730 18211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:41.632030 19417 tablet_copy_client.cc:323] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:41.636812 18211 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:41.633708 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "290a7a9eceeb4ea8a8ddb67d3e330082" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:41.633131 18136 raft_consensus.cc:2468] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:41.636320 18032 catalog_manager.cc:5671] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:41.637559 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 07a6f1b891ac4c3492ae2548ff2fc357 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:41.638145 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 07a6f1b891ac4c3492ae2548ff2fc357 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-07a6f1b891ac4c3492ae2548ff2fc357
I20260317 16:17:41.641010 18282 raft_consensus.cc:3055] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:17:41.641391 18282 raft_consensus.cc:740] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:41.654340 18282 consensus_queue.cc:260] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.655772 18282 raft_consensus.cc:3060] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:41.659885 18211 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 4.
I20260317 16:17:41.661187 18102 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:41.662113 18147 tablet_copy_source_session.cc:215] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:41.662276 19400 raft_consensus.cc:2804] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:17:41.662756 19400 raft_consensus.cc:697] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:41.664160 19400 consensus_queue.cc:237] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:41.669987 18282 raft_consensus.cc:2468] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 4.
I20260317 16:17:41.673000 18031 catalog_manager.cc:5671] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:41.676020 19417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a6f1b891ac4c3492ae2548ff2fc357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:41.701831 19417 tablet_copy_client.cc:806] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:41.702512 19417 tablet_copy_client.cc:670] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:41.715770 19414 tablet_copy_client.cc:806] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:41.716351 19414 tablet_copy_client.cc:670] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:41.717406 19420 ts_tablet_manager.cc:933] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:41.718917 19417 tablet_copy_client.cc:538] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:41.727916 19309 raft_consensus.cc:993] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:41.728600 19309 raft_consensus.cc:1081] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:41.728827 19414 tablet_copy_client.cc:538] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:41.734292 18816 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:41.734885 18816 raft_consensus.cc:493] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:41.735227 18816 raft_consensus.cc:3060] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.744400 18816 raft_consensus.cc:515] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:41.750031 19414 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:41.754081 18135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:41.754208 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfaeab6c75f94e128924d831ee8ef55b" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:41.754987 18135 raft_consensus.cc:3060] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.755045 18282 raft_consensus.cc:3055] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:41.755472 18282 raft_consensus.cc:740] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:41.757330 18282 consensus_queue.cc:260] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:41.758486 18282 raft_consensus.cc:3060] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:41.763196 18816 leader_election.cc:290] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:41.779575 18282 raft_consensus.cc:2468] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:41.781292 18782 leader_election.cc:304] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:41.787189 19387 raft_consensus.cc:2804] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:41.788090 19387 raft_consensus.cc:697] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:41.789186 19387 consensus_queue.cc:237] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:41.789459 18135 raft_consensus.cc:2468] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:41.807366 19420 tablet_copy_client.cc:323] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:41.809402 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet ab7a4647163b461f87a0a249491e566c from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:41.810086 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet ab7a4647163b461f87a0a249491e566c from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-ab7a4647163b461f87a0a249491e566c
I20260317 16:17:41.811465 19417 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:41.820452 19424 ts_tablet_manager.cc:933] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:41.833647 19424 tablet_copy_client.cc:323] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:41.742511 18031 catalog_manager.cc:5671] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 3 to 4, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 4 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:41.868639 18032 catalog_manager.cc:5671] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:41.962472 19414 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:41.970431 19414 tablet_bootstrap.cc:492] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:42.003153 19414 ts_tablet_manager.cc:1403] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.257s user 0.043s sys 0.004s
I20260317 16:17:42.027871 19414 raft_consensus.cc:359] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:42.047823 19414 raft_consensus.cc:740] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:42.092793 19414 consensus_queue.cc:260] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:42.100847 18295 tablet_copy_source_session.cc:215] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:42.104606 18137 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:42.114115 18136 raft_consensus.cc:1275] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:42.109582 19414 ts_tablet_manager.cc:1434] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.106s user 0.010s sys 0.001s
I20260317 16:17:42.120148 19420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7a4647163b461f87a0a249491e566c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:42.131103 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-a8ed2db954354e669de7909d315142aa received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:42.131870 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-a8ed2db954354e669de7909d315142aa on tablet a8ed2db954354e669de7909d315142aa with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
W20260317 16:17:42.150784 18178 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 56628.9us
W20260317 16:17:42.161063 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 265319us
I20260317 16:17:42.196295 19417 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:42.149966 19309 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260317 16:17:42.173300 19432 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260317 16:17:42.258137 18285 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:17:42.259841 18134 raft_consensus.cc:1275] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260317 16:17:42.260737 18102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 63938.7us
W20260317 16:17:42.264796 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 77046.1us
I20260317 16:17:42.267755 18208 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:17:42.270674 19417 tablet_bootstrap.cc:492] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:42.271634 19417 ts_tablet_manager.cc:1403] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.461s user 0.025s sys 0.029s
I20260317 16:17:42.274223 19267 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:42.274835 18132 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:42.303530 18284 raft_consensus.cc:1275] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:42.309860 18285 raft_consensus.cc:1275] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:42.313670 19321 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260317 16:17:42.336858 18210 raft_consensus.cc:1275] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:42.340770 19440 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260317 16:17:42.341872 19328 consensus_queue.cc:1048] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:42.306356 19417 raft_consensus.cc:359] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:42.346971 19417 raft_consensus.cc:740] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:42.347771 19417 consensus_queue.cc:260] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:42.364238 18206 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:42.382432 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet e390c786df0b48878244a08e889950a3 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:42.383082 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet e390c786df0b48878244a08e889950a3 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-e390c786df0b48878244a08e889950a3
I20260317 16:17:42.380324 19377 consensus_queue.cc:1048] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20260317 16:17:42.399477 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 55915.1us
I20260317 16:17:42.430537 19417 ts_tablet_manager.cc:1434] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.159s user 0.008s sys 0.002s
I20260317 16:17:42.434527 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-07a6f1b891ac4c3492ae2548ff2fc357 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:42.435120 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-07a6f1b891ac4c3492ae2548ff2fc357 on tablet 07a6f1b891ac4c3492ae2548ff2fc357 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:42.472348 19444 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260317 16:17:42.502714 19420 tablet_copy_client.cc:806] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:42.503585 19420 tablet_copy_client.cc:670] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:42.506083 19435 consensus_queue.cc:1048] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260317 16:17:42.500368 18222 tablet_copy_source_session.cc:215] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:42.532979 19445 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260317 16:17:42.538568 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260317 16:17:42.603252 19447 log.cc:927] Time spent T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
W20260317 16:17:42.617928 19448 log.cc:927] Time spent T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
W20260317 16:17:42.626912 19452 log.cc:927] Time spent T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260317 16:17:42.658120 18816 raft_consensus.cc:1217] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260317 16:17:42.682994 19420 tablet_copy_client.cc:538] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260317 16:17:42.691606 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 146707us
I20260317 16:17:42.721508 18816 raft_consensus.cc:1217] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
W20260317 16:17:42.719897 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 47907.9us
W20260317 16:17:42.742856 18781 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 47945.7us
I20260317 16:17:42.747315 19424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e390c786df0b48878244a08e889950a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:42.752492 19420 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
W20260317 16:17:42.798066 19418 log.cc:927] Time spent T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: Append to log took a long time: real 0.151s user 0.000s sys 0.003s
I20260317 16:17:42.804157 19420 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:42.805181 19420 tablet_bootstrap.cc:492] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:42.806128 19420 ts_tablet_manager.cc:1403] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.054s user 0.026s sys 0.006s
I20260317 16:17:42.811124 19420 raft_consensus.cc:359] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:42.813022 19420 raft_consensus.cc:740] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:42.814143 19420 consensus_queue.cc:260] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:42.823016 19420 ts_tablet_manager.cc:1434] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20260317 16:17:42.827381 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-ab7a4647163b461f87a0a249491e566c received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:42.827890 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-ab7a4647163b461f87a0a249491e566c on tablet ab7a4647163b461f87a0a249491e566c with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:43.104784 19424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:57316 (request call id 398) took 1304 ms. Trace:
I20260317 16:17:43.105293 19424 rpcz_store.cc:276] 0317 16:17:41.800746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:17:41.805744 (+ 4998us) service_pool.cc:225] Handling call
0317 16:17:41.821310 (+ 15566us) ts_tablet_manager.cc:934] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
0317 16:17:42.745922 (+924612us) tablet_copy_client.cc:368] Tablet copy session begun
0317 16:17:42.803071 (+ 57149us) tablet_metadata.cc:641] Metadata flushed
0317 16:17:42.803352 (+ 281us) tablet_copy_client.cc:497] Wrote new tablet metadata
0317 16:17:43.103856 (+300504us) tablet_copy_client.cc:418] Wrote new consensus metadata
0317 16:17:43.104149 (+ 293us) tablet_replica.cc:206] Creating consensus instance
0317 16:17:43.104442 (+ 293us) ts_tablet_manager.cc:989] Replica registered.
0317 16:17:43.104709 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":396,"tablet-copy.queue_time_us":3059,"thread_start_us":12799,"threads_started":2}
I20260317 16:17:43.109686 19424 tablet_copy_client.cc:806] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:43.110742 19424 tablet_copy_client.cc:670] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:43.120432 19442 raft_consensus.cc:1064] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:43.125865 19442 consensus_queue.cc:237] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:43.136729 19400 raft_consensus.cc:1064] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:43.139986 19400 consensus_queue.cc:237] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:43.156292 18812 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:43.160554 18282 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:43.163070 19400 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:43.166074 19434 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:43.166707 18206 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260317 16:17:43.173270 18252 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:17:43.176240 19428 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:43.177142 18817 raft_consensus.cc:1217] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:43.196931 19424 tablet_copy_client.cc:538] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:43.268079 19424 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
W20260317 16:17:43.254138 18781 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 59783us
I20260317 16:17:43.287406 18816 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:43.289724 19432 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260317 16:17:43.298233 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 48130.5us
I20260317 16:17:43.300750 18208 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:43.315343 19390 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:43.328488 19321 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.331032 18210 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:43.331822 18210 raft_consensus.cc:606] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:43.341563 18282 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.347036 18134 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:43.385818 18206 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.399350 18281 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:43.400177 18281 raft_consensus.cc:606] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:43.404966 18130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:43.405820 18130 raft_consensus.cc:606] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:43.407842 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260317 16:17:43.410470 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260317 16:17:43.412088 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 66401.7us
I20260317 16:17:43.425789 19309 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:43.430130 18816 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.470294 18031 catalog_manager.cc:5671] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:43.481756 18207 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.474460 19457 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
W20260317 16:17:43.517390 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 96864.7us
I20260317 16:17:43.501526 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:43.530318 19424 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:43.543642 19424 tablet_bootstrap.cc:492] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:43.559725 19424 ts_tablet_manager.cc:1403] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.292s user 0.032s sys 0.035s
I20260317 16:17:43.565402 19424 raft_consensus.cc:359] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:43.566407 19424 raft_consensus.cc:740] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:43.559463 18130 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.585167 18812 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.647241 19457 raft_consensus.cc:993] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:43.653184 19445 raft_consensus.cc:1081] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:43.660662 19424 consensus_queue.cc:260] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:43.663342 18130 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:43.664129 18130 raft_consensus.cc:493] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:43.664963 19424 ts_tablet_manager.cc:1434] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.103s user 0.004s sys 0.002s
I20260317 16:17:43.667995 18130 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:43.670115 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-e390c786df0b48878244a08e889950a3 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:43.670738 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-e390c786df0b48878244a08e889950a3 on tablet e390c786df0b48878244a08e889950a3 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:43.673095 19432 raft_consensus.cc:1064] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:43.676159 19432 consensus_queue.cc:237] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:43.678473 18032 catalog_manager.cc:5671] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.684494 19262 raft_consensus.cc:993] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:43.685032 19262 raft_consensus.cc:1081] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:43.687422 18286 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:43.688498 18286 raft_consensus.cc:493] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:43.688887 18286 raft_consensus.cc:3060] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:43.698557 18134 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:43.700461 19432 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:43.701443 18210 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:43.703330 19457 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:43.722081 18130 raft_consensus.cc:515] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:43.725065 18130 leader_election.cc:290] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:43.726092 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:43.734572 18282 raft_consensus.cc:3055] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:43.734964 18282 raft_consensus.cc:740] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:43.735744 18812 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:43.735724 18282 consensus_queue.cc:260] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:43.736886 18282 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:43.733033 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:43.737442 18210 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:43.739749 19457 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:43.736459 18281 consensus_queue.cc:237] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:43.761737 18210 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:43.753515 18136 consensus_queue.cc:237] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:43.770180 18206 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260317 16:17:43.771272 18102 leader_election.cc:304] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:43.771656 19432 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:43.772789 19483 raft_consensus.cc:2804] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:43.773289 19483 raft_consensus.cc:697] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:43.773932 18282 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:43.774185 19483 consensus_queue.cc:237] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:43.777604 18816 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:43.767287 18817 raft_consensus.cc:1275] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260317 16:17:43.788543 19309 consensus_queue.cc:1048] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:43.794602 19445 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.806252 19251 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.817087 18817 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.818995 18281 raft_consensus.cc:1275] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:43.808413 18812 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.782384 19428 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260317 16:17:43.808012 18136 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.731508 18286 raft_consensus.cc:515] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:43.834321 18130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:43.834506 18286 leader_election.cc:290] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:43.835260 18130 raft_consensus.cc:3060] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:43.836233 18206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1897e8c1602b4aeda93d9b8207b3ac45" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:43.836858 18206 raft_consensus.cc:3055] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:43.837141 18206 raft_consensus.cc:740] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:43.837963 18206 consensus_queue.cc:260] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:43.841451 19400 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.848531 18210 raft_consensus.cc:2955] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.862439 18817 raft_consensus.cc:1217] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:43.865844 18206 raft_consensus.cc:3060] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:43.868858 18034 catalog_manager.cc:5671] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.876039 18130 raft_consensus.cc:2468] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:43.907680 18249 leader_election.cc:304] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:43.910553 18207 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.913043 19432 raft_consensus.cc:2804] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:43.917604 19483 consensus_queue.cc:1048] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:43.927233 18206 raft_consensus.cc:2468] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:43.927930 19432 raft_consensus.cc:697] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
W20260317 16:17:43.920444 18102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 61317.4us
I20260317 16:17:43.929045 19432 consensus_queue.cc:237] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:43.920281 18812 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.956864 18032 catalog_manager.cc:5671] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:43.947346 18281 raft_consensus.cc:2955] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.956573 18031 catalog_manager.cc:5671] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:43.993167 18282 consensus_queue.cc:237] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:44.003991 18812 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:44.010915 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet a8ed2db954354e669de7909d315142aa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:44.028932 19428 raft_consensus.cc:993] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:44.030992 19483 raft_consensus.cc:1081] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:44.029016 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 07a6f1b891ac4c3492ae2548ff2fc357 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:44.037637 19445 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:44.037456 18207 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:44.039963 18207 raft_consensus.cc:493] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:44.041476 18207 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:44.084597 19457 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.092561 18812 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.106290 18206 raft_consensus.cc:1275] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:44.108389 18207 raft_consensus.cc:515] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:44.110388 18207 leader_election.cc:290] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:44.145397 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet a8ed2db954354e669de7909d315142aa with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:44.134078 18034 catalog_manager.cc:5671] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 4 to 5, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:44.162753 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet ab7a4647163b461f87a0a249491e566c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:44.175295 19499 tablet_replica.cc:333] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:44.177932 19499 raft_consensus.cc:2243] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:44.163574 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:44.178754 18282 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:44.184394 18130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77f02a05d0ee4e73bafe3055cfcf02f2" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:44.185237 18130 raft_consensus.cc:3055] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:44.185657 18130 raft_consensus.cc:740] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:44.187778 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a6f1b891ac4c3492ae2548ff2fc357 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:44.188989 18032 catalog_manager.cc:5671] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:44.191576 19499 raft_consensus.cc:2272] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:44.186396 18130 consensus_queue.cc:260] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:44.200199 18130 raft_consensus.cc:3060] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:44.207532 19442 consensus_queue.cc:1048] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:44.208918 18206 raft_consensus.cc:1275] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:44.210682 19495 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:44.211800 19499 ts_tablet_manager.cc:1916] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:44.224272 18207 raft_consensus.cc:2955] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.234619 18035 catalog_manager.cc:5671] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
W20260317 16:17:44.248173 18782 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 49953.5us
I20260317 16:17:44.250417 19500 tablet_replica.cc:333] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:44.269528 18282 raft_consensus.cc:2468] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:44.257063 19500 raft_consensus.cc:2243] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:44.271507 18178 leader_election.cc:304] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:44.271970 19500 raft_consensus.cc:2272] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:44.283516 18282 raft_consensus.cc:1275] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:44.286633 19377 raft_consensus.cc:2804] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:44.292308 19482 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:44.293675 19377 raft_consensus.cc:697] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:44.295003 19377 consensus_queue.cc:237] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:44.305639 19500 ts_tablet_manager.cc:1916] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:44.316501 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet ab7a4647163b461f87a0a249491e566c with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:44.324440 19478 raft_consensus.cc:1064] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:44.258912 18130 raft_consensus.cc:2468] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:44.327283 19478 consensus_queue.cc:237] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:44.336978 19499 ts_tablet_manager.cc:1929] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20260317 16:17:44.339385 19499 log.cc:1199] T a8ed2db954354e669de7909d315142aa P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/a8ed2db954354e669de7909d315142aa
I20260317 16:17:44.341457 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet a8ed2db954354e669de7909d315142aa (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:44.341917 19499 tablet_replica.cc:333] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:44.343461 19499 raft_consensus.cc:2243] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:44.345008 19499 raft_consensus.cc:2272] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:44.349083 19499 ts_tablet_manager.cc:1916] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:44.367879 19500 ts_tablet_manager.cc:1929] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260317 16:17:44.368409 19500 log.cc:1199] T 07a6f1b891ac4c3492ae2548ff2fc357 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/07a6f1b891ac4c3492ae2548ff2fc357
I20260317 16:17:44.381515 18283 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:44.381500 18130 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:44.383942 19377 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:44.386754 18817 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:44.392369 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 07a6f1b891ac4c3492ae2548ff2fc357 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:44.396029 19478 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:44.402977 19377 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:44.420147 18035 catalog_manager.cc:5671] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 2 to 3, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:44.430838 19478 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.433833 18283 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.449054 18817 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.389356 19499 ts_tablet_manager.cc:1929] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:44.454134 19499 log.cc:1199] T ab7a4647163b461f87a0a249491e566c P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/ab7a4647163b461f87a0a249491e566c
I20260317 16:17:44.471995 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet ab7a4647163b461f87a0a249491e566c (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:44.474229 18130 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.476852 18034 catalog_manager.cc:5671] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.506973 18130 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:44.511023 19390 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:44.511957 18208 consensus_queue.cc:237] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:44.528612 18207 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:44.530375 19390 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:44.535136 18817 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:44.535724 18136 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:44.536962 19377 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:44.542856 19262 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:44.550063 19377 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.553906 18817 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.567297 18130 raft_consensus.cc:2955] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.586826 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e390c786df0b48878244a08e889950a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:44.592638 18035 catalog_manager.cc:5671] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:44.623181 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet e390c786df0b48878244a08e889950a3 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:44.625448 19515 tablet_replica.cc:333] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:44.626541 19515 raft_consensus.cc:2243] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:44.627480 19515 raft_consensus.cc:2272] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:44.630966 19515 ts_tablet_manager.cc:1916] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:44.643795 19515 ts_tablet_manager.cc:1929] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:44.644199 19515 log.cc:1199] T e390c786df0b48878244a08e889950a3 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/e390c786df0b48878244a08e889950a3
I20260317 16:17:44.645745 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet e390c786df0b48878244a08e889950a3 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:44.745996 18283 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:44.747326 19478 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:44.757182 18130 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:44.759028 19478 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:44.970377 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:45.011122 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:45.011691 18208 raft_consensus.cc:606] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:45.014432 18283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:45.014912 18283 raft_consensus.cc:606] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:45.017498 18283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:45.018088 18283 raft_consensus.cc:606] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:45.021288 18136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:45.021901 18136 raft_consensus.cc:606] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:45.023485 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260317 16:17:45.024675 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:45.025230 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:45.130079 19478 raft_consensus.cc:993] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:45.130558 19478 raft_consensus.cc:1081] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:45.133859 18283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:45.134701 18283 raft_consensus.cc:493] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:45.135123 18283 raft_consensus.cc:3060] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.141234 18283 raft_consensus.cc:515] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.142974 18283 leader_election.cc:290] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:45.144002 18130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:45.144205 18207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f120096f51b4b3a992048069e2ae182" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:45.144788 18130 raft_consensus.cc:3060] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.144856 18207 raft_consensus.cc:3055] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:45.146337 18207 raft_consensus.cc:740] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:45.147132 18207 consensus_queue.cc:260] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.148322 18207 raft_consensus.cc:3060] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.152348 18130 raft_consensus.cc:2468] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:45.153609 18249 leader_election.cc:304] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:45.154469 19457 raft_consensus.cc:2804] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:45.155011 19457 raft_consensus.cc:697] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:45.155189 18207 raft_consensus.cc:2468] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:45.156059 19457 consensus_queue.cc:237] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.166477 18034 catalog_manager.cc:5671] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:45.174376 19495 raft_consensus.cc:993] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:45.175017 19434 raft_consensus.cc:1081] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:45.176455 18208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:45.177047 18208 raft_consensus.cc:493] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:45.177367 18208 raft_consensus.cc:3060] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.183523 18208 raft_consensus.cc:515] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.184922 18208 leader_election.cc:290] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:45.185918 18130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:45.186017 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1e740c77f5f4cb6916938c76fd04455" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:45.186549 18130 raft_consensus.cc:3055] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:45.186609 18283 raft_consensus.cc:3060] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.186867 18130 raft_consensus.cc:740] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:45.187635 18130 consensus_queue.cc:260] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.188755 18130 raft_consensus.cc:3060] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.191844 18283 raft_consensus.cc:2468] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:45.193153 18178 leader_election.cc:304] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:45.193843 18130 raft_consensus.cc:2468] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:45.193882 19478 raft_consensus.cc:2804] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:45.194537 19478 raft_consensus.cc:697] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:45.195477 19478 consensus_queue.cc:237] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.205991 18034 catalog_manager.cc:5671] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:45.347882 19390 raft_consensus.cc:993] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:45.348405 19457 raft_consensus.cc:1081] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:45.350077 18130 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:45.350586 18130 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:45.350975 18130 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:45.356071 18130 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.357292 18130 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 3 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:45.358166 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:45.358208 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:45.359091 18208 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:45.359148 18283 raft_consensus.cc:3055] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:45.359645 18283 raft_consensus.cc:740] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:45.360329 18283 consensus_queue.cc:260] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.361333 18283 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:45.365728 18208 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 3.
I20260317 16:17:45.366861 18102 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:45.367503 19495 raft_consensus.cc:2804] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:45.368134 19495 raft_consensus.cc:697] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:45.369061 18283 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 3.
I20260317 16:17:45.368947 19495 consensus_queue.cc:237] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:45.379814 18034 catalog_manager.cc:5671] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 2 to 3, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 3 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:45.453676 19457 raft_consensus.cc:993] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:45.454254 19457 raft_consensus.cc:1081] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:45.455755 18817 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:45.456208 18817 raft_consensus.cc:493] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:45.456465 18817 raft_consensus.cc:3060] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.461555 18817 raft_consensus.cc:515] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:45.464233 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:45.464697 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55ecf327ed214b02bef87b8c05d54c26" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:45.464843 18136 raft_consensus.cc:3060] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.465268 18283 raft_consensus.cc:3055] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:45.465574 18283 raft_consensus.cc:740] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:45.464461 18817 leader_election.cc:290] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:45.466288 18283 consensus_queue.cc:260] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:45.467392 18283 raft_consensus.cc:3060] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:45.471951 18136 raft_consensus.cc:2468] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:45.473107 18779 leader_election.cc:304] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:45.473850 18283 raft_consensus.cc:2468] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:45.474471 19501 raft_consensus.cc:2804] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:45.475075 19501 raft_consensus.cc:697] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:45.475961 19501 consensus_queue.cc:237] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:45.486464 18034 catalog_manager.cc:5671] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:45.674731 18283 raft_consensus.cc:1275] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:45.681558 19377 consensus_queue.cc:1048] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:45.700644 18130 raft_consensus.cc:1275] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:45.704151 19377 consensus_queue.cc:1048] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:45.725844 18281 consensus_queue.cc:237] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:45.735666 18130 raft_consensus.cc:1275] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:45.738370 19390 consensus_queue.cc:1048] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:45.754186 18207 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:45.756208 19457 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:45.756495 18130 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:45.760143 19457 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260317 16:17:45.767611 18249 consensus_peers.cc:597] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94f2b31d108e461d932c2477e0f78fd0. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:45.772531 18207 raft_consensus.cc:1275] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:45.773314 18206 raft_consensus.cc:1275] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:45.774228 19495 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:45.782948 19390 consensus_queue.cc:1048] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:45.796957 19390 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:45.802726 18206 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:45.837447 18281 raft_consensus.cc:1275] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:45.840718 19523 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:45.901048 18130 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:45.896299 18283 consensus_queue.cc:237] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:45.919744 18032 catalog_manager.cc:5671] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:45.923105 18130 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260317 16:17:45.924019 18249 consensus_peers.cc:597] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1897e8c1602b4aeda93d9b8207b3ac45. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:45.924528 19440 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:45.925410 18136 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:45.928407 18208 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:45.934250 19390 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:45.949795 19390 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:45.983156 19535 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:45.995661 18102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 43547.9us
I20260317 16:17:46.014751 18208 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.023639 18136 consensus_queue.cc:237] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.026422 18032 catalog_manager.cc:5671] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:46.038022 18137 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.045264 18283 raft_consensus.cc:1275] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:46.060923 19440 consensus_queue.cc:1048] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:46.063979 18210 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:46.066567 18100 consensus_peers.cc:597] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c85ea9795a2c43a38f1a9c3f3741db36. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:46.071687 19495 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:46.067118 18283 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:46.068176 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:46.088742 19523 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:46.108495 19434 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.109738 18283 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.117153 18210 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.132519 18208 consensus_queue.cc:237] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.140856 18032 catalog_manager.cc:5671] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:46.169909 18175 consensus_peers.cc:597] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96d59cc0e53747428c5ff7f5178f11ca. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:46.170787 18283 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:46.172953 18137 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:46.173713 18137 raft_consensus.cc:606] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:46.175088 19478 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:46.177533 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:46.177734 18137 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:46.178128 18208 raft_consensus.cc:606] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:46.179471 19478 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:46.180648 18281 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:46.181159 18281 raft_consensus.cc:606] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:46.182402 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260317 16:17:46.184576 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:46.185994 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:46.190191 19377 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.192397 18281 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.197440 18137 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:46.211151 18035 catalog_manager.cc:5671] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:46.282351 19556 ts_tablet_manager.cc:933] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:46.298167 19556 tablet_copy_client.cc:323] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:46.300390 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 94f2b31d108e461d932c2477e0f78fd0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:46.301095 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 94f2b31d108e461d932c2477e0f78fd0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-94f2b31d108e461d932c2477e0f78fd0
I20260317 16:17:46.310696 18295 tablet_copy_source_session.cc:215] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:46.314078 19556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f2b31d108e461d932c2477e0f78fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:46.326025 19434 raft_consensus.cc:993] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:46.327699 19523 raft_consensus.cc:1081] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:46.329478 18210 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:46.330041 18210 raft_consensus.cc:493] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:46.330394 18210 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:46.338153 18210 raft_consensus.cc:515] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:46.340683 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:46.341363 18136 raft_consensus.cc:3055] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:46.341883 18136 raft_consensus.cc:740] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:46.343164 18136 consensus_queue.cc:260] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:46.344264 18136 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:46.342319 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:46.347021 18283 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:46.352442 19556 tablet_copy_client.cc:806] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:46.353080 19556 tablet_copy_client.cc:670] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:46.353546 18210 leader_election.cc:290] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:46.367199 18136 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:46.369414 18175 leader_election.cc:304] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:46.370275 19377 raft_consensus.cc:2804] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:46.370734 19377 raft_consensus.cc:697] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:46.371528 19556 tablet_copy_client.cc:538] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:46.357741 18283 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:46.371536 19377 consensus_queue.cc:237] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:46.382390 19556 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:46.397540 18035 catalog_manager.cc:5671] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 2 to 3, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:46.417200 19556 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:46.418805 19556 tablet_bootstrap.cc:492] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:46.419512 19556 ts_tablet_manager.cc:1403] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.008s
I20260317 16:17:46.424216 19556 raft_consensus.cc:359] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.425060 19556 raft_consensus.cc:740] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:46.425745 19556 consensus_queue.cc:260] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.428365 19556 ts_tablet_manager.cc:1434] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 16:17:46.430459 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-94f2b31d108e461d932c2477e0f78fd0 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:46.430968 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-94f2b31d108e461d932c2477e0f78fd0 on tablet 94f2b31d108e461d932c2477e0f78fd0 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:46.454977 19556 ts_tablet_manager.cc:933] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:46.458027 19556 tablet_copy_client.cc:323] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:46.460105 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 1897e8c1602b4aeda93d9b8207b3ac45 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:46.460592 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 1897e8c1602b4aeda93d9b8207b3ac45 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-1897e8c1602b4aeda93d9b8207b3ac45
I20260317 16:17:46.461681 19478 raft_consensus.cc:993] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:46.462188 19377 raft_consensus.cc:1081] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:46.465018 18283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:46.465626 18283 raft_consensus.cc:493] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:46.465975 18283 raft_consensus.cc:3060] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:46.468858 18295 tablet_copy_source_session.cc:215] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:46.472091 18283 raft_consensus.cc:515] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:46.472738 19556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1897e8c1602b4aeda93d9b8207b3ac45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:46.474560 18130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:46.475342 18130 raft_consensus.cc:3060] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:46.475507 18283 leader_election.cc:290] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:46.476562 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85f478d86a0949f381f0d88a657ba4e5" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:46.477492 18208 raft_consensus.cc:3055] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:46.479110 18208 raft_consensus.cc:740] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:46.480050 18208 consensus_queue.cc:260] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:46.482570 18130 raft_consensus.cc:2468] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:46.485559 18249 leader_election.cc:304] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:46.484397 19561 ts_tablet_manager.cc:933] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:46.487315 19390 raft_consensus.cc:2804] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:46.489831 19390 raft_consensus.cc:697] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:46.490939 19390 consensus_queue.cc:237] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:46.496302 19556 tablet_copy_client.cc:806] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:46.496920 19556 tablet_copy_client.cc:670] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:46.497021 18208 raft_consensus.cc:3060] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:46.499790 19561 tablet_copy_client.cc:323] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:46.501370 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet c85ea9795a2c43a38f1a9c3f3741db36 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:46.501879 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet c85ea9795a2c43a38f1a9c3f3741db36 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-c85ea9795a2c43a38f1a9c3f3741db36
I20260317 16:17:46.503304 19556 tablet_copy_client.cc:538] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:46.518639 18147 tablet_copy_source_session.cc:215] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:46.522151 19556 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:46.525285 19561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ea9795a2c43a38f1a9c3f3741db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:46.527892 18208 raft_consensus.cc:2468] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:46.542042 18035 catalog_manager.cc:5671] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:46.564517 19561 tablet_copy_client.cc:806] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:46.565146 19561 tablet_copy_client.cc:670] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:46.565806 19556 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:46.567754 19556 tablet_bootstrap.cc:492] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:46.569290 19556 ts_tablet_manager.cc:1403] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.011s
I20260317 16:17:46.574198 19561 tablet_copy_client.cc:538] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:46.575551 19556 raft_consensus.cc:359] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.576617 19556 raft_consensus.cc:740] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:46.577301 19556 consensus_queue.cc:260] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.580228 19556 ts_tablet_manager.cc:1434] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260317 16:17:46.582899 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-1897e8c1602b4aeda93d9b8207b3ac45 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:46.583361 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-1897e8c1602b4aeda93d9b8207b3ac45 on tablet 1897e8c1602b4aeda93d9b8207b3ac45 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:46.587082 19556 ts_tablet_manager.cc:933] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:46.601706 19561 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:46.610247 19556 tablet_copy_client.cc:323] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:46.615909 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 96d59cc0e53747428c5ff7f5178f11ca from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:46.616516 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 96d59cc0e53747428c5ff7f5178f11ca from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-96d59cc0e53747428c5ff7f5178f11ca
I20260317 16:17:46.634105 18222 tablet_copy_source_session.cc:215] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:46.634805 19561 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:46.635778 19561 tablet_bootstrap.cc:492] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:46.636495 19561 ts_tablet_manager.cc:1403] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.008s
I20260317 16:17:46.639415 19556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d59cc0e53747428c5ff7f5178f11ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:46.639539 19561 raft_consensus.cc:359] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.640383 19561 raft_consensus.cc:740] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:46.641016 19561 consensus_queue.cc:260] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.688757 19561 ts_tablet_manager.cc:1434] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.052s user 0.010s sys 0.000s
I20260317 16:17:46.693939 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-c85ea9795a2c43a38f1a9c3f3741db36 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:46.694434 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-c85ea9795a2c43a38f1a9c3f3741db36 on tablet c85ea9795a2c43a38f1a9c3f3741db36 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:46.702608 19556 tablet_copy_client.cc:806] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:46.707991 19556 tablet_copy_client.cc:670] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:46.712131 19556 tablet_copy_client.cc:538] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:46.719700 19568 raft_consensus.cc:993] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:46.720275 19390 raft_consensus.cc:1081] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:46.725991 19556 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:46.727837 18137 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:46.729111 18137 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:46.729492 18137 raft_consensus.cc:3060] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:46.739326 18137 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:46.742192 18817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:46.742910 18817 raft_consensus.cc:3060] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:46.747118 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:46.747756 18281 raft_consensus.cc:3055] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:46.748029 18281 raft_consensus.cc:740] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:46.748709 18281 consensus_queue.cc:260] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:46.749747 18281 raft_consensus.cc:3060] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:46.750108 18817 raft_consensus.cc:2468] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:46.751258 18100 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:46.752207 19523 raft_consensus.cc:2804] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:46.752871 19523 raft_consensus.cc:697] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:46.753738 19523 consensus_queue.cc:237] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:46.761116 18281 raft_consensus.cc:2468] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:46.765267 18137 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:46.772827 19556 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:46.775868 19556 tablet_bootstrap.cc:492] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:46.779340 19556 ts_tablet_manager.cc:1403] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.053s user 0.023s sys 0.007s
I20260317 16:17:46.781042 18035 catalog_manager.cc:5671] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:46.784354 19556 raft_consensus.cc:359] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.785398 19556 raft_consensus.cc:740] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:46.791124 19556 consensus_queue.cc:260] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:46.793469 19556 ts_tablet_manager.cc:1434] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260317 16:17:46.799542 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-96d59cc0e53747428c5ff7f5178f11ca received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:46.800033 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-96d59cc0e53747428c5ff7f5178f11ca on tablet 96d59cc0e53747428c5ff7f5178f11ca with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:46.822312 18130 raft_consensus.cc:1275] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:46.831653 19377 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:46.869971 18283 raft_consensus.cc:1275] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:46.871367 19478 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:46.905973 18817 raft_consensus.cc:1217] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260317 16:17:46.910946 18208 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:46.912806 19534 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:46.918409 18130 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:46.919771 19534 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:46.959254 18817 raft_consensus.cc:1217] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260317 16:17:47.046504 18817 raft_consensus.cc:1217] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:47.148336 18817 raft_consensus.cc:1217] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:47.151386 18816 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:47.153308 19580 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:47.172302 18282 raft_consensus.cc:1275] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:47.173695 19580 consensus_queue.cc:1048] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:47.259999 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:47.309835 18210 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:47.310375 18210 raft_consensus.cc:606] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:47.312947 18283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:47.313603 18283 raft_consensus.cc:606] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:47.316186 18130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:47.316717 18130 raft_consensus.cc:606] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:47.317799 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260317 16:17:47.319111 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:47.319725 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:47.352471 19568 raft_consensus.cc:1064] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:47.353194 19568 raft_consensus.cc:1068] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Service unavailable: leader transfer in progress
I20260317 16:17:47.365123 19568 raft_consensus.cc:993] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:47.365578 19390 raft_consensus.cc:1081] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:47.367159 18136 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47378
I20260317 16:17:47.367686 18136 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:47.368031 18136 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.382367 18136 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:47.384828 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:47.385504 18210 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.387460 18136 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:47.389209 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:47.389886 18282 raft_consensus.cc:3055] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:47.390247 18282 raft_consensus.cc:740] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:47.391069 18282 consensus_queue.cc:260] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:47.393882 18282 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.401768 18282 raft_consensus.cc:2468] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:47.402310 18210 raft_consensus.cc:2468] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:47.403106 18103 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:47.403891 19580 raft_consensus.cc:2804] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:47.404359 19580 raft_consensus.cc:697] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:47.405828 19580 consensus_queue.cc:237] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:47.424600 18035 catalog_manager.cc:5671] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:47.478508 19523 raft_consensus.cc:1064] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:47.479017 19523 raft_consensus.cc:1068] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Service unavailable: leader transfer in progress
I20260317 16:17:47.507819 19572 raft_consensus.cc:993] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:47.508389 19377 raft_consensus.cc:1081] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:47.509743 18282 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:47.510278 18282 raft_consensus.cc:493] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:47.510795 18282 raft_consensus.cc:3060] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.518465 19579 raft_consensus.cc:1064] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:47.521728 19579 consensus_queue.cc:237] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.529691 18282 raft_consensus.cc:515] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.531276 18816 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:47.532274 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:47.532863 18210 raft_consensus.cc:3055] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:47.533152 18210 raft_consensus.cc:740] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:47.533859 18210 consensus_queue.cc:260] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.534250 18817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b2a4d319ba9480b9c950e12b7329101" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:47.534798 18817 raft_consensus.cc:3060] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.534947 18210 raft_consensus.cc:3060] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.539697 18130 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:47.533167 18208 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:47.541106 19579 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:47.542156 19568 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:47.543685 18210 raft_consensus.cc:2468] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:47.546198 18251 leader_election.cc:304] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:47.546994 19579 raft_consensus.cc:2804] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:47.547487 19579 raft_consensus.cc:697] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:47.550320 19568 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:47.548738 19579 consensus_queue.cc:237] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.551532 18817 raft_consensus.cc:2468] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:47.553282 18282 leader_election.cc:290] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:47.560174 19478 raft_consensus.cc:1064] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:47.562790 19478 consensus_queue.cc:237] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.566038 19568 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.570369 18817 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.596755 18130 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.599236 18812 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:47.598708 18208 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.602605 19594 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:47.618911 18136 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:47.620543 19572 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:47.620677 18034 catalog_manager.cc:5671] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.618703 18035 catalog_manager.cc:5671] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:47.643814 18281 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:47.646159 19377 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:47.808189 19377 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.815774 18812 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.817844 18283 consensus_queue.cc:237] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.842367 19523 raft_consensus.cc:993] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:47.842895 19523 raft_consensus.cc:1081] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:47.844997 18206 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:47.845582 18206 raft_consensus.cc:493] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:47.845953 18206 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.863088 18130 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.869059 18817 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:47.869678 18204 raft_consensus.cc:1275] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:47.870669 19568 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:47.872157 19597 consensus_queue.cc:1048] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260317 16:17:47.872300 18178 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 58133.9us
I20260317 16:17:47.893873 18206 raft_consensus.cc:515] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:47.880875 18034 catalog_manager.cc:5671] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.896327 18206 leader_election.cc:290] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:47.900969 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:47.901566 18136 raft_consensus.cc:3055] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:47.901878 18136 raft_consensus.cc:740] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:47.903425 18136 consensus_queue.cc:260] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:47.904505 18136 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.911907 19595 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.914520 18136 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:47.916071 18285 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:47.922214 19580 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:47.924993 18175 leader_election.cc:304] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:47.928038 19478 raft_consensus.cc:2804] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:47.928561 19478 raft_consensus.cc:697] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:47.929540 19478 consensus_queue.cc:237] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:47.929065 18817 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.935633 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 1897e8c1602b4aeda93d9b8207b3ac45 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:47.951042 18031 catalog_manager.cc:5671] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:47.963048 18285 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.965967 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 1897e8c1602b4aeda93d9b8207b3ac45 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:47.974182 18204 consensus_queue.cc:237] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:47.975510 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:47.976349 18282 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:47.988768 18817 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:47.990427 18206 raft_consensus.cc:2955] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:47.992923 18130 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:47.996145 19377 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:48.002089 19478 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260317 16:17:48.006599 19605 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.020153 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 96d59cc0e53747428c5ff7f5178f11ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:48.015260 18282 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:48.023200 19608 tablet_replica.cc:333] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:48.024307 19608 raft_consensus.cc:2243] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:48.024107 18817 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.025199 19608 raft_consensus.cc:2272] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:48.088768 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 56355.8us
I20260317 16:17:48.043792 18034 catalog_manager.cc:5671] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:48.039490 18134 raft_consensus.cc:2955] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.053099 18816 raft_consensus.cc:3060] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Advancing to term 2
I20260317 16:17:48.109189 18812 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:48.110949 19568 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:48.040832 18210 raft_consensus.cc:1275] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260317 16:17:48.092343 18102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 57160.7us
I20260317 16:17:48.135020 19597 consensus_queue.cc:1048] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:48.162580 18210 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:48.169996 19608 ts_tablet_manager.cc:1916] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:48.172607 18210 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:48.173235 18210 raft_consensus.cc:606] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Received request to transfer leadership
I20260317 16:17:48.175338 19611 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:48.181370 18816 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:48.184589 18034 catalog_manager.cc:5671] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:48.192688 18206 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:48.193815 18206 raft_consensus.cc:606] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Received request to transfer leadership
W20260317 16:17:48.195115 18021 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for c85ea9795a2c43a38f1a9c3f3741db36 already in progress [suppressed 2 similar messages]
I20260317 16:17:48.203213 19608 ts_tablet_manager.cc:1929] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260317 16:17:48.203872 19608 log.cc:1199] T 1897e8c1602b4aeda93d9b8207b3ac45 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/1897e8c1602b4aeda93d9b8207b3ac45
I20260317 16:17:48.206014 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 1897e8c1602b4aeda93d9b8207b3ac45 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:48.206396 19604 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:48.209082 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 96d59cc0e53747428c5ff7f5178f11ca with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:48.228816 19434 raft_consensus.cc:1064] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:48.230998 19604 consensus_queue.cc:1175] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "021d78130690452ba094f4a392fb4378" responder_term: 2 status { last_received { term: 2 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 2 index: 4 } }
I20260317 16:17:48.233814 19434 consensus_queue.cc:237] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.237543 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:48.238152 18208 raft_consensus.cc:606] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Received request to transfer leadership
W20260317 16:17:48.243738 18083 auto_rebalancer.cc:798] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260317 16:17:48.244166 18083 auto_rebalancer.cc:806] Could not move replica: Remote error: Service unavailable: leadership transfer for c85ea9795a2c43a38f1a9c3f3741db36 already in progress
W20260317 16:17:48.244419 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for c85ea9795a2c43a38f1a9c3f3741db36 already in progress
I20260317 16:17:48.271346 19617 tablet_replica.cc:333] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:48.274987 18817 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:48.275813 18281 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:48.277158 19523 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:48.278811 18204 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:48.279270 19612 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:48.281363 19434 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:48.291021 19617 raft_consensus.cc:2243] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:48.295073 19617 raft_consensus.cc:2272] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:48.300024 19617 ts_tablet_manager.cc:1916] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:48.311028 19523 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.310720 18281 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.316951 18204 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.321352 19617 ts_tablet_manager.cc:1929] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:48.322019 19617 log.cc:1199] T 96d59cc0e53747428c5ff7f5178f11ca P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/96d59cc0e53747428c5ff7f5178f11ca
I20260317 16:17:48.324016 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 96d59cc0e53747428c5ff7f5178f11ca (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:48.337782 18035 catalog_manager.cc:5671] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.338960 18812 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.352064 18817 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Advancing to term 2
I20260317 16:17:48.366216 18134 consensus_queue.cc:237] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.381222 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:48.388116 18817 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:48.388304 18812 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260317 16:17:48.389470 19478 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:48.389928 18281 raft_consensus.cc:1275] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260317 16:17:48.389673 19580 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:48.398500 19609 consensus_queue.cc:1048] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:17:48.407074 19594 raft_consensus.cc:1064] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:48.408001 19594 raft_consensus.cc:1068] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Service unavailable: leader transfer in progress
I20260317 16:17:48.408042 19434 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.412887 18283 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.419909 18817 raft_consensus.cc:2955] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.425369 18281 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:48.430671 19605 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:48.436772 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 94f2b31d108e461d932c2477e0f78fd0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:48.439389 18035 catalog_manager.cc:5671] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.462635 19594 raft_consensus.cc:993] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:48.463148 19594 raft_consensus.cc:1081] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:48.464751 18282 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:48.466187 18282 raft_consensus.cc:493] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:48.467885 18136 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:48.466569 18282 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:48.476513 19377 raft_consensus.cc:1064] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:48.476291 19594 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260317 16:17:48.479646 19377 raft_consensus.cc:1068] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Service unavailable: leader transfer in progress
I20260317 16:17:48.480854 18282 raft_consensus.cc:515] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:48.483719 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:48.487561 18282 leader_election.cc:290] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:48.491256 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:48.491796 18208 raft_consensus.cc:606] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:48.493135 18204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ea9795a2c43a38f1a9c3f3741db36" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:48.493757 18204 raft_consensus.cc:3055] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:48.494098 18204 raft_consensus.cc:740] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:48.500293 18204 consensus_queue.cc:260] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:48.501957 18204 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:48.503446 18136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:48.504034 18136 raft_consensus.cc:606] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:48.506079 19594 raft_consensus.cc:993] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:48.508392 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 94f2b31d108e461d932c2477e0f78fd0 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:48.506580 19594 raft_consensus.cc:1081] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:48.510425 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:48.511368 18208 raft_consensus.cc:606] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:48.515053 18204 raft_consensus.cc:2468] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 3.
I20260317 16:17:48.516247 18282 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:48.516798 18282 raft_consensus.cc:493] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:48.518272 18282 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.516362 18283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:48.519682 18251 leader_election.cc:304] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters: 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:48.520603 19568 raft_consensus.cc:2804] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:48.521221 19568 raft_consensus.cc:697] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:48.522114 19568 consensus_queue.cc:237] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:48.530227 18283 raft_consensus.cc:606] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:48.532186 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260317 16:17:48.532598 19625 tablet_replica.cc:333] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:48.533818 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:48.534636 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:48.534917 19625 raft_consensus.cc:2243] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:48.535125 18282 raft_consensus.cc:515] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.535647 19625 raft_consensus.cc:2272] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:48.539952 18282 leader_election.cc:290] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:48.540565 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:48.541323 18812 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.543452 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:48.544054 18208 raft_consensus.cc:3055] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:48.544435 18208 raft_consensus.cc:740] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:48.546304 18208 consensus_queue.cc:260] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.547919 18208 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.558641 18208 raft_consensus.cc:2468] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:48.560286 18812 raft_consensus.cc:2468] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:48.561128 18251 leader_election.cc:304] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:48.564469 19568 raft_consensus.cc:2804] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:48.561141 19625 ts_tablet_manager.cc:1916] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:48.566298 19434 raft_consensus.cc:993] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:48.566957 19434 raft_consensus.cc:1081] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:48.563014 18034 catalog_manager.cc:5671] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 2 to 3, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:48.568809 18282 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33384
I20260317 16:17:48.569343 18282 raft_consensus.cc:493] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:48.569666 18282 raft_consensus.cc:3060] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.572436 19568 raft_consensus.cc:697] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:48.587057 19568 consensus_queue.cc:237] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.593282 18282 raft_consensus.cc:515] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.596613 18282 leader_election.cc:290] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:48.600476 19625 ts_tablet_manager.cc:1929] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20260317 16:17:48.600986 19625 log.cc:1199] T 94f2b31d108e461d932c2477e0f78fd0 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/94f2b31d108e461d932c2477e0f78fd0
I20260317 16:17:48.602619 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 94f2b31d108e461d932c2477e0f78fd0 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:48.603600 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:48.605150 18136 raft_consensus.cc:3055] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:48.606040 18136 raft_consensus.cc:740] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:48.607153 19594 raft_consensus.cc:993] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:48.606808 18136 consensus_queue.cc:260] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.607611 19594 raft_consensus.cc:1081] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:48.608251 18136 raft_consensus.cc:3060] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.609975 18130 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47376
I20260317 16:17:48.610543 18130 raft_consensus.cc:493] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:48.610916 18130 raft_consensus.cc:3060] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.613448 18817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fe426def0d4f378a44ce5ec328f2e0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:48.614012 18817 raft_consensus.cc:3060] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.618103 18136 raft_consensus.cc:2468] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:48.619072 18130 raft_consensus.cc:515] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.619549 18249 leader_election.cc:304] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:48.623291 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:48.623751 18130 leader_election.cc:290] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:48.624691 18208 raft_consensus.cc:3055] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:48.625145 18208 raft_consensus.cc:740] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:48.626104 18208 consensus_queue.cc:260] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.627265 18208 raft_consensus.cc:3060] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.629835 18817 raft_consensus.cc:2468] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:48.631356 18817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96d59cc0e53747428c5ff7f5178f11ca" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:48.631914 18817 raft_consensus.cc:3060] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.634362 19595 raft_consensus.cc:2804] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:48.643616 18817 raft_consensus.cc:2468] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:48.644704 19595 raft_consensus.cc:697] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:48.644852 18100 leader_election.cc:304] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:48.646565 19595 consensus_queue.cc:237] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.645942 19434 raft_consensus.cc:2804] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:48.648169 19434 raft_consensus.cc:697] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:48.649223 19434 consensus_queue.cc:237] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.655571 18034 catalog_manager.cc:5671] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:48.656912 18208 raft_consensus.cc:2468] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:48.667251 18031 catalog_manager.cc:5671] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:48.678586 18034 catalog_manager.cc:5671] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:48.806546 19595 raft_consensus.cc:993] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:48.807359 19597 raft_consensus.cc:1081] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:48.809120 18817 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:48.809684 18817 raft_consensus.cc:493] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:48.810007 18817 raft_consensus.cc:3060] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.815495 18817 raft_consensus.cc:515] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.817092 18817 leader_election.cc:290] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:48.817979 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:48.818132 18283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6aca2493a994ed888cd663ddcf97582" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:48.818454 18208 raft_consensus.cc:3060] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.818825 18283 raft_consensus.cc:3055] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:48.819094 18283 raft_consensus.cc:740] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:48.819743 18283 consensus_queue.cc:260] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.820711 18283 raft_consensus.cc:3060] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:48.823195 18208 raft_consensus.cc:2468] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:48.824288 18781 leader_election.cc:304] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:48.825013 19501 raft_consensus.cc:2804] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:48.825402 19501 raft_consensus.cc:697] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:48.825534 18283 raft_consensus.cc:2468] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:48.826156 19501 consensus_queue.cc:237] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.842738 18034 catalog_manager.cc:5671] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:48.908064 18208 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:48.909415 19597 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:48.919708 18817 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Advancing to term 3
I20260317 16:17:48.924139 18130 raft_consensus.cc:3060] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:48.932838 18817 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:48.932839 18130 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:48.936822 19629 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:48.937556 19597 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:48.945318 19595 raft_consensus.cc:1064] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:48.948673 19595 consensus_queue.cc:237] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:48.958801 18817 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:48.960012 19629 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:48.960649 18136 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:48.964752 19629 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:48.972036 18204 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:48.975210 19595 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:48.981701 19629 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.984213 18130 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.985754 18204 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:48.994655 18817 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:49.002267 18034 catalog_manager.cc:5671] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:49.020503 18282 consensus_queue.cc:237] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:49.028649 18130 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:49.030567 19568 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:49.034094 18208 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.038040 19568 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.049402 18204 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.052119 19434 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.055231 18817 raft_consensus.cc:1275] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:49.059510 19568 consensus_queue.cc:1048] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:49.073405 19595 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:49.086385 18812 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.086133 18130 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:49.087986 19629 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.091706 18816 raft_consensus.cc:1275] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.095571 18817 raft_consensus.cc:2955] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:49.097944 19611 consensus_queue.cc:1048] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.133090 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet c85ea9795a2c43a38f1a9c3f3741db36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:49.139314 18034 catalog_manager.cc:5671] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 5 to 6, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 3 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:49.173424 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet c85ea9795a2c43a38f1a9c3f3741db36 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:49.177011 19633 tablet_replica.cc:333] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:49.178629 19633 raft_consensus.cc:2243] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:49.179368 19633 raft_consensus.cc:2272] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:49.183200 19633 ts_tablet_manager.cc:1916] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:49.191756 18130 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.192008 18816 raft_consensus.cc:1275] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.193337 19597 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.194078 19629 consensus_queue.cc:1048] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.199999 19633 ts_tablet_manager.cc:1929] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260317 16:17:49.200690 19633 log.cc:1199] T c85ea9795a2c43a38f1a9c3f3741db36 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/c85ea9795a2c43a38f1a9c3f3741db36
I20260317 16:17:49.202991 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet c85ea9795a2c43a38f1a9c3f3741db36 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:49.306329 18204 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.308020 19602 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.332904 18282 raft_consensus.cc:1275] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:49.336093 19501 consensus_queue.cc:1048] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:49.522044 18136 consensus_queue.cc:237] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:49.528712 18204 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:17:49.528837 18282 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260317 16:17:49.529114 18100 consensus_peers.cc:597] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 290a7a9eceeb4ea8a8ddb67d3e330082. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:49.530695 19434 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:49.532954 19611 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:49.542999 19434 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.544497 18283 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.549371 18208 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.557487 18283 consensus_queue.cc:237] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:49.566356 18035 catalog_manager.cc:5671] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 4 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:49.577409 18208 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:49.578974 19629 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:49.580641 18136 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260317 16:17:49.588491 18249 consensus_peers.cc:597] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 519bc447dbc1455c844945ba63c96914. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:49.593811 19595 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:49.596122 19597 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.608670 18130 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.603987 18208 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.613466 18136 consensus_queue.cc:237] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:49.627674 18034 catalog_manager.cc:5671] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:49.640801 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260317 16:17:49.652535 18100 consensus_peers.cc:597] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eb8a46e02034bd199afdbe545379351. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:49.652894 18282 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:49.657300 18204 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:49.662520 19611 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:49.677971 19643 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:49.734220 19611 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.741815 18283 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.751907 18204 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.757958 18281 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:49.758569 18281 raft_consensus.cc:606] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:49.757373 18210 consensus_queue.cc:237] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:49.767342 18031 catalog_manager.cc:5671] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
W20260317 16:17:49.770972 18175 consensus_peers.cc:597] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cf03ff443ed41d3bb50d992c1f34372. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:49.772661 18136 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:49.774204 19377 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:49.776191 18281 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:49.778292 19377 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:49.784554 18281 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:49.786559 18281 raft_consensus.cc:606] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:49.798096 19594 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.800545 18136 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.801762 18282 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.803298 18283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:49.804003 18283 raft_consensus.cc:606] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:49.807132 18130 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:49.807647 18130 raft_consensus.cc:606] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:49.809115 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260317 16:17:49.812114 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:49.813093 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:49.833851 18031 catalog_manager.cc:5671] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:49.899772 19597 raft_consensus.cc:993] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:49.900503 19568 raft_consensus.cc:1081] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:49.902482 18204 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57500
I20260317 16:17:49.903169 18204 raft_consensus.cc:493] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:49.903558 18204 raft_consensus.cc:3060] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:49.910490 18204 raft_consensus.cc:515] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:49.912788 18204 leader_election.cc:290] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:49.913249 18130 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:49.913823 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9246439ff23646f8abdf7bba7f3a9172" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:49.913956 18130 raft_consensus.cc:3060] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:49.914319 18281 raft_consensus.cc:3055] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:49.914588 18281 raft_consensus.cc:740] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:49.915331 18281 consensus_queue.cc:260] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:49.916361 18281 raft_consensus.cc:3060] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:49.926254 18130 raft_consensus.cc:2468] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:49.926270 18281 raft_consensus.cc:2468] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:49.927878 18175 leader_election.cc:304] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:49.929627 19377 raft_consensus.cc:2804] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:49.930145 19377 raft_consensus.cc:697] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:49.931775 19377 consensus_queue.cc:237] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:49.943758 18031 catalog_manager.cc:5671] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:49.960345 19568 raft_consensus.cc:993] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:49.960830 19597 raft_consensus.cc:1081] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:49.962255 18204 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57500
I20260317 16:17:49.962812 18204 raft_consensus.cc:493] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:49.963201 18204 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:49.977371 18204 raft_consensus.cc:515] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:49.983985 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:49.984573 18281 raft_consensus.cc:3055] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:49.984875 18281 raft_consensus.cc:740] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:49.985869 18281 consensus_queue.cc:260] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:49.987517 18281 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:49.989099 19657 ts_tablet_manager.cc:933] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:49.997862 19657 tablet_copy_client.cc:323] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:49.979502 19656 ts_tablet_manager.cc:933] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:50.000974 18204 leader_election.cc:290] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:50.002753 19629 raft_consensus.cc:993] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:50.003829 19595 raft_consensus.cc:1081] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:50.006984 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 290a7a9eceeb4ea8a8ddb67d3e330082 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:50.014294 18204 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57500
I20260317 16:17:50.016928 18204 raft_consensus.cc:493] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:50.018275 18204 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:50.004623 18281 raft_consensus.cc:2468] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:50.034746 18178 leader_election.cc:304] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:50.035880 19605 raft_consensus.cc:2804] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:50.036484 19605 raft_consensus.cc:697] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:50.037511 19605 consensus_queue.cc:237] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:50.056598 18031 catalog_manager.cc:5671] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 2 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:50.064118 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:50.065450 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 290a7a9eceeb4ea8a8ddb67d3e330082 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-290a7a9eceeb4ea8a8ddb67d3e330082
I20260317 16:17:50.078923 18137 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:50.075384 19656 tablet_copy_client.cc:323] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:50.085803 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 519bc447dbc1455c844945ba63c96914 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:50.089213 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 519bc447dbc1455c844945ba63c96914 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-519bc447dbc1455c844945ba63c96914
I20260317 16:17:50.090083 18137 raft_consensus.cc:2468] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 2.
I20260317 16:17:50.101475 18295 tablet_copy_source_session.cc:215] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:50.101583 18147 tablet_copy_source_session.cc:215] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:50.077867 18204 raft_consensus.cc:515] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.105799 18204 leader_election.cc:290] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Requested vote from peers 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:50.106001 19657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290a7a9eceeb4ea8a8ddb67d3e330082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:50.106572 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:50.107189 18281 raft_consensus.cc:3055] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:50.107476 18281 raft_consensus.cc:740] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:50.108242 18281 consensus_queue.cc:260] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.108847 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07c5d59b5b404eec925d5c4681953edb" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:50.109364 18815 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:50.109483 18281 raft_consensus.cc:3060] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:50.117534 19656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519bc447dbc1455c844945ba63c96914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:50.119940 18815 raft_consensus.cc:2468] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:50.123755 18175 leader_election.cc:304] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:50.117494 18281 raft_consensus.cc:2468] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:50.126813 19377 raft_consensus.cc:2804] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:50.127470 19377 raft_consensus.cc:697] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:50.128561 19377 consensus_queue.cc:237] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.154418 18031 catalog_manager.cc:5671] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 2 to 3, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:50.158586 19656 tablet_copy_client.cc:806] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:50.159271 19656 tablet_copy_client.cc:670] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:50.165417 19657 tablet_copy_client.cc:806] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:50.166085 19657 tablet_copy_client.cc:670] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:50.176313 19657 tablet_copy_client.cc:538] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:50.178895 19656 tablet_copy_client.cc:538] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:50.191783 19657 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:50.215102 19656 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:50.226403 19663 ts_tablet_manager.cc:933] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:50.230275 19662 ts_tablet_manager.cc:933] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:50.232381 19663 tablet_copy_client.cc:323] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:50.234493 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 1cf03ff443ed41d3bb50d992c1f34372 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:50.235417 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 1cf03ff443ed41d3bb50d992c1f34372 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-1cf03ff443ed41d3bb50d992c1f34372
I20260317 16:17:50.244141 19662 tablet_copy_client.cc:323] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:50.247812 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 0eb8a46e02034bd199afdbe545379351 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:50.248358 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 0eb8a46e02034bd199afdbe545379351 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-0eb8a46e02034bd199afdbe545379351
I20260317 16:17:50.249325 18222 tablet_copy_source_session.cc:215] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:50.253229 19663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf03ff443ed41d3bb50d992c1f34372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:50.258215 19656 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:50.259240 19656 tablet_bootstrap.cc:492] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:50.259935 19656 ts_tablet_manager.cc:1403] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.007s
I20260317 16:17:50.264303 19656 raft_consensus.cc:359] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.265180 19656 raft_consensus.cc:740] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:50.266055 19657 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:50.265865 19656 consensus_queue.cc:260] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.267308 19657 tablet_bootstrap.cc:492] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:50.268003 19657 ts_tablet_manager.cc:1403] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.077s user 0.022s sys 0.011s
I20260317 16:17:50.270177 19656 ts_tablet_manager.cc:1434] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260317 16:17:50.273077 19657 raft_consensus.cc:359] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.273953 19657 raft_consensus.cc:740] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:50.275331 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-519bc447dbc1455c844945ba63c96914 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:50.275801 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-519bc447dbc1455c844945ba63c96914 on tablet 519bc447dbc1455c844945ba63c96914 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:50.278556 18147 tablet_copy_source_session.cc:215] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:50.279707 19663 tablet_copy_client.cc:806] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:50.281662 19663 tablet_copy_client.cc:670] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:50.281937 19657 consensus_queue.cc:260] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.283367 19662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb8a46e02034bd199afdbe545379351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:50.285406 19657 ts_tablet_manager.cc:1434] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260317 16:17:50.299594 19663 tablet_copy_client.cc:538] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:50.307413 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-290a7a9eceeb4ea8a8ddb67d3e330082 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:50.308315 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-290a7a9eceeb4ea8a8ddb67d3e330082 on tablet 290a7a9eceeb4ea8a8ddb67d3e330082 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:50.316846 19662 tablet_copy_client.cc:806] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:50.317449 19662 tablet_copy_client.cc:670] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:50.317786 19663 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:50.323004 19662 tablet_copy_client.cc:538] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:50.338786 19662 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:50.349081 19663 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:50.351810 19663 tablet_bootstrap.cc:492] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:50.352463 19663 ts_tablet_manager.cc:1403] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.015s
I20260317 16:17:50.363886 19663 raft_consensus.cc:359] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.364771 19663 raft_consensus.cc:740] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:50.365440 19663 consensus_queue.cc:260] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.368738 19663 ts_tablet_manager.cc:1434] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20260317 16:17:50.374255 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-1cf03ff443ed41d3bb50d992c1f34372 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:50.374909 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-1cf03ff443ed41d3bb50d992c1f34372 on tablet 1cf03ff443ed41d3bb50d992c1f34372 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:50.381460 19662 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:50.382550 19662 tablet_bootstrap.cc:492] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:50.383309 19662 ts_tablet_manager.cc:1403] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.002s
I20260317 16:17:50.385742 19611 raft_consensus.cc:993] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:50.386286 19611 raft_consensus.cc:1081] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:50.390013 18816 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57886
I20260317 16:17:50.390564 18816 raft_consensus.cc:493] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:50.390933 18816 raft_consensus.cc:3060] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:50.388012 19662 raft_consensus.cc:359] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.395375 19662 raft_consensus.cc:740] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:50.396095 19662 consensus_queue.cc:260] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:50.398154 19662 ts_tablet_manager.cc:1434] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.014s user 0.001s sys 0.004s
I20260317 16:17:50.399977 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-0eb8a46e02034bd199afdbe545379351 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:50.400504 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-0eb8a46e02034bd199afdbe545379351 on tablet 0eb8a46e02034bd199afdbe545379351 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:50.399956 18816 raft_consensus.cc:515] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.407038 18816 leader_election.cc:290] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:50.407835 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:50.408113 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d88548519a0f473680897b8a7fc1e9b6" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:50.408458 18210 raft_consensus.cc:3060] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:50.409963 18137 raft_consensus.cc:3055] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:50.410357 18137 raft_consensus.cc:740] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:50.412468 18137 consensus_queue.cc:260] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.413897 18137 raft_consensus.cc:3060] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:50.417156 18210 raft_consensus.cc:2468] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:50.419327 18781 leader_election.cc:304] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:50.420265 19501 raft_consensus.cc:2804] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:50.420725 19501 raft_consensus.cc:697] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:50.421474 19501 consensus_queue.cc:237] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.428603 18137 raft_consensus.cc:2468] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:50.441955 18815 raft_consensus.cc:1217] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260317 16:17:50.447438 18034 catalog_manager.cc:5671] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:50.451371 18137 raft_consensus.cc:1275] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:50.455488 19594 consensus_queue.cc:1048] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:50.482146 18286 raft_consensus.cc:1275] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:50.483505 19594 consensus_queue.cc:1048] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:50.549675 18815 raft_consensus.cc:1217] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20260317 16:17:50.550741 18286 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:50.552073 19377 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:50.574497 18816 raft_consensus.cc:1217] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:50.577667 18815 raft_consensus.cc:1275] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:50.582372 19605 consensus_queue.cc:1048] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:17:50.598115 18282 raft_consensus.cc:1275] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:50.599550 19605 consensus_queue.cc:1048] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:50.619653 18137 raft_consensus.cc:1275] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:17:50.628014 19594 consensus_queue.cc:1048] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:50.712983 18815 raft_consensus.cc:1217] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:50.835140 18137 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:50.836616 19501 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:50.857139 18204 raft_consensus.cc:1275] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:50.859470 19501 consensus_queue.cc:1048] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:50.883436 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:50.910687 19568 raft_consensus.cc:1064] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:50.913188 19568 consensus_queue.cc:237] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:50.919412 18816 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:50.919554 18204 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:50.921058 19595 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:50.922134 19597 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:50.923382 18137 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:50.959864 19595 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:51.003336 19629 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.017210 18131 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.019141 18204 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:51.019763 18204 raft_consensus.cc:606] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:51.024066 18204 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:51.020782 18816 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.025583 18204 raft_consensus.cc:606] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:51.026638 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260317 16:17:51.029361 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:51.030144 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:51.029259 18204 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.039927 18035 catalog_manager.cc:5671] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:51.075033 19674 raft_consensus.cc:993] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:51.075551 19674 raft_consensus.cc:1081] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:51.077556 18281 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33388
I20260317 16:17:51.078068 18281 raft_consensus.cc:493] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:51.078380 18281 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:51.098716 18281 raft_consensus.cc:515] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:51.110993 18281 leader_election.cc:290] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 4 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:51.112272 18204 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:51.112627 18137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a773cd94be84692a95e55317c98a95c" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:51.112938 18204 raft_consensus.cc:3055] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:17:51.113169 18137 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:51.113250 18204 raft_consensus.cc:740] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:51.114210 18204 consensus_queue.cc:260] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:51.115396 18204 raft_consensus.cc:3060] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:51.121340 18137 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 4.
I20260317 16:17:51.123055 18249 leader_election.cc:304] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:51.124087 19595 raft_consensus.cc:2804] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:17:51.124679 19595 raft_consensus.cc:697] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 4 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:51.125490 19595 consensus_queue.cc:237] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:51.128724 18281 consensus_queue.cc:237] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:51.129868 18204 raft_consensus.cc:2468] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 4.
I20260317 16:17:51.139896 19604 raft_consensus.cc:1064] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:51.143114 18816 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:51.143337 19604 consensus_queue.cc:237] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:51.144796 19629 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:51.154145 18210 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:51.155639 19629 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:51.159860 18816 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.164554 19595 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.168996 18204 raft_consensus.cc:2955] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.170539 19611 raft_consensus.cc:1064] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:51.177690 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 519bc447dbc1455c844945ba63c96914 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:51.173588 19611 consensus_queue.cc:237] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:51.181674 19605 raft_consensus.cc:1064] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:51.182327 19605 raft_consensus.cc:1068] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Service unavailable: leader transfer in progress
I20260317 16:17:51.183802 18034 catalog_manager.cc:5671] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.187940 18210 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:51.194969 18817 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:51.198069 19643 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:51.200208 18817 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260317 16:17:51.201334 18286 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:51.203028 18281 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260317 16:17:51.215634 18032 catalog_manager.cc:5671] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 3 to 4, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 4 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } }
I20260317 16:17:51.229810 19697 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:51.232681 19697 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:51.232798 19699 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260317 16:17:51.237433 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 519bc447dbc1455c844945ba63c96914 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:51.239367 19701 tablet_replica.cc:333] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:51.240818 19701 raft_consensus.cc:2243] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:51.241580 19701 raft_consensus.cc:2272] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:51.251451 18102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 59795.8us
I20260317 16:17:51.253473 18208 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260317 16:17:51.256031 18103 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 47110.4us
I20260317 16:17:51.271008 19700 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:51.274075 19699 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:51.283422 19701 ts_tablet_manager.cc:1916] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:51.308933 19697 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.318914 19604 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.324261 18281 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.324960 18286 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.325888 18816 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.327965 18817 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.411728 19701 ts_tablet_manager.cc:1929] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260317 16:17:51.412269 19701 log.cc:1199] T 519bc447dbc1455c844945ba63c96914 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/519bc447dbc1455c844945ba63c96914
I20260317 16:17:51.413938 18210 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.417289 18204 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.417328 18031 catalog_manager.cc:5671] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.438877 18032 catalog_manager.cc:5671] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.444716 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 519bc447dbc1455c844945ba63c96914 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:51.473308 18134 consensus_queue.cc:237] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:51.478099 18816 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:51.480096 19611 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:51.481169 18286 raft_consensus.cc:1275] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:51.472680 18136 consensus_queue.cc:237] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:51.482477 19611 consensus_queue.cc:1048] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:51.490705 18210 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260317 16:17:51.491884 19611 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:51.492115 18815 raft_consensus.cc:1275] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260317 16:17:51.492307 19643 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.493602 19694 consensus_queue.cc:1048] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:51.494195 18815 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.504212 19694 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.506959 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 0eb8a46e02034bd199afdbe545379351 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:51.516160 18817 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.521596 18204 raft_consensus.cc:2955] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.520000 18286 raft_consensus.cc:2955] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.530972 18035 catalog_manager.cc:5671] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.542909 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eb8a46e02034bd199afdbe545379351 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:51.556636 19708 tablet_replica.cc:333] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:51.559489 19674 raft_consensus.cc:993] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:51.560962 19674 raft_consensus.cc:1081] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:51.562561 19708 raft_consensus.cc:2243] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:51.563618 19708 raft_consensus.cc:2272] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:51.564481 18131 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47376
I20260317 16:17:51.565100 18131 raft_consensus.cc:493] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:51.565449 18131 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:51.575412 19708 ts_tablet_manager.cc:1916] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:51.576035 18134 raft_consensus.cc:1275] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:17:51.577435 19597 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:51.582911 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 290a7a9eceeb4ea8a8ddb67d3e330082 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:51.565061 18031 catalog_manager.cc:5671] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 5 to 6, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 4 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:51.602378 18131 raft_consensus.cc:515] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:51.615875 18131 leader_election.cc:290] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:51.616786 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:51.617805 18208 raft_consensus.cc:3055] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:51.618247 18208 raft_consensus.cc:740] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:51.619272 18208 consensus_queue.cc:260] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:51.619693 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:51.620267 18281 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:51.620858 18208 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:51.628687 18208 raft_consensus.cc:2468] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:51.630808 19708 ts_tablet_manager.cc:1929] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:51.631259 19708 log.cc:1199] T 0eb8a46e02034bd199afdbe545379351 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/0eb8a46e02034bd199afdbe545379351
I20260317 16:17:51.631786 18281 raft_consensus.cc:2468] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:51.632992 18103 leader_election.cc:304] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:51.633049 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 0eb8a46e02034bd199afdbe545379351 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:51.635389 19611 raft_consensus.cc:2804] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:51.635890 19611 raft_consensus.cc:697] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:51.636963 19611 consensus_queue.cc:237] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:51.642314 18210 raft_consensus.cc:1275] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:17:51.645078 19595 consensus_queue.cc:1048] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:51.647351 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 290a7a9eceeb4ea8a8ddb67d3e330082 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:51.685832 19712 tablet_replica.cc:333] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:51.687062 19712 raft_consensus.cc:2243] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:51.692597 19712 raft_consensus.cc:2272] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:51.703111 19712 ts_tablet_manager.cc:1916] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:51.706070 18035 catalog_manager.cc:5671] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:51.724182 18134 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:51.724619 18134 raft_consensus.cc:606] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Received request to transfer leadership
I20260317 16:17:51.725159 19712 ts_tablet_manager.cc:1929] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20260317 16:17:51.725831 19712 log.cc:1199] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/290a7a9eceeb4ea8a8ddb67d3e330082
I20260317 16:17:51.727413 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 290a7a9eceeb4ea8a8ddb67d3e330082 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:51.732229 18134 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:51.732715 18134 raft_consensus.cc:606] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Received request to transfer leadership
I20260317 16:17:51.736020 18134 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:51.736483 18134 raft_consensus.cc:606] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Received request to transfer leadership
W20260317 16:17:51.741101 18083 auto_rebalancer.cc:798] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260317 16:17:51.741453 18083 auto_rebalancer.cc:806] Could not move replica: Remote error: Service unavailable: leadership transfer for 1cf03ff443ed41d3bb50d992c1f34372 already in progress
W20260317 16:17:51.741650 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 1cf03ff443ed41d3bb50d992c1f34372 already in progress
I20260317 16:17:52.059914 18816 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Advancing to term 2
I20260317 16:17:52.072650 18816 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:52.074810 19604 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:52.092360 19611 raft_consensus.cc:1064] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:52.093096 19611 raft_consensus.cc:1068] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Service unavailable: leader transfer in progress
I20260317 16:17:52.146064 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:52.230973 18206 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:52.232625 19698 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:52.254978 19643 raft_consensus.cc:993] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:52.255650 19643 raft_consensus.cc:1081] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:52.257594 18208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:52.266196 18286 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "519bc447dbc1455c844945ba63c96914"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:52.266971 18286 raft_consensus.cc:606] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:52.271867 18286 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:52.273576 19643 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:52.276175 18208 raft_consensus.cc:493] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:52.276023 18134 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:52.276726 18208 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:52.277034 18134 raft_consensus.cc:606] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Received request to transfer leadership to TS 021d78130690452ba094f4a392fb4378
I20260317 16:17:52.278456 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260317 16:17:52.280298 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:52.281364 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:52.287211 18208 raft_consensus.cc:515] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:52.297180 18208 leader_election.cc:290] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:52.307284 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:52.308811 19643 raft_consensus.cc:1064] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:52.306430 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf03ff443ed41d3bb50d992c1f34372" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:52.308863 18134 raft_consensus.cc:3055] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:52.312024 18134 raft_consensus.cc:740] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:52.313112 18134 consensus_queue.cc:260] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:52.314713 18134 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:52.339102 18134 raft_consensus.cc:2468] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:52.341104 18175 leader_election.cc:304] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d; no voters: 021d78130690452ba094f4a392fb4378
I20260317 16:17:52.341940 19722 raft_consensus.cc:2804] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:52.342444 19722 raft_consensus.cc:697] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
W20260317 16:17:52.342370 19643 raft_consensus.cc:1068] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Illegal state: Replica 94edf207909c4f8990bfe29ed1a87a6d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.343604 19722 consensus_queue.cc:237] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:52.361653 18034 catalog_manager.cc:5671] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 2 to 3, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:52.556142 19698 raft_consensus.cc:993] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:52.556844 19698 raft_consensus.cc:1081] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Signalling peer 021d78130690452ba094f4a392fb4378 to start an election
I20260317 16:17:52.558578 18281 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1"
dest_uuid: "021d78130690452ba094f4a392fb4378"
from {username='slave'} at 127.0.0.1:33384
I20260317 16:17:52.559129 18281 raft_consensus.cc:493] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:52.559497 18281 raft_consensus.cc:3060] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:52.564827 18281 raft_consensus.cc:515] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:52.566390 18281 leader_election.cc:290] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:52.567188 18134 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:52.567497 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb31cad8b88240acb9c101027ea444c1" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:52.567793 18134 raft_consensus.cc:3055] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:52.568022 18815 raft_consensus.cc:3060] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:52.568145 18134 raft_consensus.cc:740] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:52.569015 18134 consensus_queue.cc:260] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:52.571902 18134 raft_consensus.cc:3060] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:52.577121 18815 raft_consensus.cc:2468] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:52.577754 18134 raft_consensus.cc:2468] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:52.578490 18249 leader_election.cc:304] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:52.579397 19568 raft_consensus.cc:2804] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:52.579859 19568 raft_consensus.cc:697] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:52.580492 19568 consensus_queue.cc:237] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:52.591255 18034 catalog_manager.cc:5671] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 reported cstate change: term changed from 1 to 2, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 021d78130690452ba094f4a392fb4378 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:52.756538 18815 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Advancing to term 3
I20260317 16:17:52.762121 18815 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:52.763522 19594 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:52.771291 19594 raft_consensus.cc:1064] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
W20260317 16:17:52.771744 19594 raft_consensus.cc:1068] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: Unable to promote non-voter 381cc8622e1f4a53a7ff2b23ed2bcd38: Illegal state: Leader has not yet committed an operation in its own term
I20260317 16:17:52.788182 19629 raft_consensus.cc:993] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:52.788699 19568 raft_consensus.cc:1081] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:52.790117 18208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "519bc447dbc1455c844945ba63c96914"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57500
I20260317 16:17:52.790681 18208 raft_consensus.cc:493] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:52.791102 18208 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:52.797667 18208 raft_consensus.cc:515] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:52.799391 18208 leader_election.cc:290] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Requested vote from peers 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:52.800124 18281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:52.800305 18815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "519bc447dbc1455c844945ba63c96914" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:52.800926 18815 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:52.802459 18281 raft_consensus.cc:3055] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:52.802958 18281 raft_consensus.cc:740] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:52.803766 18281 consensus_queue.cc:260] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:52.807276 18281 raft_consensus.cc:3060] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:52.807461 18815 raft_consensus.cc:2468] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
I20260317 16:17:52.808696 18175 leader_election.cc:304] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:52.809636 19594 raft_consensus.cc:2804] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:52.810348 19594 raft_consensus.cc:697] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:52.811335 19594 consensus_queue.cc:237] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:52.816807 18286 consensus_queue.cc:237] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:52.817512 18281 raft_consensus.cc:2468] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 3.
W20260317 16:17:52.826284 18249 consensus_peers.cc:597] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85f478d86a0949f381f0d88a657ba4e5. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:52.827732 18134 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:52.829186 19568 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:52.828773 18034 catalog_manager.cc:5671] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 2 to 3, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:52.832875 18208 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:52.834347 19735 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:52.839001 19595 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.841101 18134 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.858927 18137 consensus_queue.cc:237] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:52.859462 18034 catalog_manager.cc:5671] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:52.869931 18208 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.871367 18281 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:52.872310 18207 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:52.874225 19611 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:17:52.875195 19698 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:17:52.891645 18100 consensus_peers.cc:597] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75ecbd9bfe9e4fe4b1f5146d5029d1c0. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:52.904682 19611 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.907428 18208 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.911820 18281 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.917874 18286 raft_consensus.cc:3060] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:52.919243 18134 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:52.922380 19722 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:52.928762 18134 consensus_queue.cc:237] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:52.932474 18286 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:52.931115 18035 catalog_manager.cc:5671] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:52.934650 19722 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:52.954433 18281 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 16:17:52.955070 18208 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 16:17:52.955726 18100 consensus_peers.cc:597] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c9d82ef74c442299f4a3cf04a5bb2d7. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:52.960096 19643 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:17:52.962507 19643 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260317 16:17:52.973523 18815 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:52.984684 19739 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260317 16:17:53.139178 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 183914us
I20260317 16:17:53.164073 19759 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.173439 18133 raft_consensus.cc:1275] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:53.180876 18207 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.210995 19595 consensus_queue.cc:1048] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260317 16:17:53.214394 19594 raft_consensus.cc:1064] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:53.217700 19594 consensus_queue.cc:237] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.226564 18286 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.230396 18208 consensus_queue.cc:237] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.233757 18281 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:53.234941 18812 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:53.235940 19753 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260317 16:17:53.250917 19746 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:53.252971 18134 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:53.254321 18034 catalog_manager.cc:5671] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index -1 to 2, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.258711 19746 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:53.268707 19722 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.281216 18812 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.283214 18134 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.305145 18281 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:53.304350 18286 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.308408 18032 catalog_manager.cc:5671] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 2 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.320561 19744 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260317 16:17:53.345392 18175 consensus_peers.cc:597] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77f02a05d0ee4e73bafe3055cfcf02f2. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:53.349779 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:53.359835 19767 ts_tablet_manager.cc:933] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:53.362082 19767 tablet_copy_client.cc:323] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:53.364166 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 85f478d86a0949f381f0d88a657ba4e5 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:53.364777 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 85f478d86a0949f381f0d88a657ba4e5 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-85f478d86a0949f381f0d88a657ba4e5
I20260317 16:17:53.364846 18134 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:53.384169 19744 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:53.386051 18295 tablet_copy_source_session.cc:215] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:53.392634 19767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f478d86a0949f381f0d88a657ba4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:53.396498 18812 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:53.427677 18286 raft_consensus.cc:1275] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:53.444988 19767 tablet_copy_client.cc:806] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:53.445953 19767 tablet_copy_client.cc:670] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
W20260317 16:17:53.459429 19770 log.cc:927] Time spent T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260317 16:17:53.462723 18281 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc"
dest_uuid: "021d78130690452ba094f4a392fb4378"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:41546
I20260317 16:17:53.463371 18281 raft_consensus.cc:606] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:53.464048 18208 consensus_queue.cc:237] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.471068 19744 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.475034 19766 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:17:53.477730 18813 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:53.481971 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:53.482637 18208 raft_consensus.cc:606] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:53.485122 19767 tablet_copy_client.cc:538] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:53.488858 18206 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e390c786df0b48878244a08e889950a3"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:53.489734 18206 raft_consensus.cc:606] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:53.491882 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260317 16:17:53.492949 19765 consensus_queue.cc:1048] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:17:53.493592 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:53.495193 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:53.498509 19722 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:17:53.498904 18134 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.498904 18281 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.518652 19767 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:53.524027 18286 raft_consensus.cc:1275] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:53.526583 19744 consensus_queue.cc:1048] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260317 16:17:53.548892 19772 ts_tablet_manager.cc:933] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:53.553772 19771 ts_tablet_manager.cc:933] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:53.563594 19766 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.573238 18285 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.580585 19772 tablet_copy_client.cc:323] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:53.572877 18032 catalog_manager.cc:5671] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:53.592711 18812 raft_consensus.cc:2955] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.600891 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:53.601544 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-8c9d82ef74c442299f4a3cf04a5bb2d7
I20260317 16:17:53.624421 18147 tablet_copy_source_session.cc:215] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:53.627202 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 1cf03ff443ed41d3bb50d992c1f34372 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:53.633481 19772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:53.639345 19771 tablet_copy_client.cc:323] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:53.647783 19767 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:53.647118 18034 catalog_manager.cc:5671] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:53.648741 19767 tablet_bootstrap.cc:492] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:53.649443 19767 ts_tablet_manager.cc:1403] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.131s user 0.020s sys 0.012s
I20260317 16:17:53.655097 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:53.655673 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-75ecbd9bfe9e4fe4b1f5146d5029d1c0
I20260317 16:17:53.663522 18147 tablet_copy_source_session.cc:215] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:53.665417 19772 tablet_copy_client.cc:806] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:53.666145 19772 tablet_copy_client.cc:670] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:53.667487 19771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:53.667306 19767 raft_consensus.cc:359] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.668288 19767 raft_consensus.cc:740] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:53.669063 19767 consensus_queue.cc:260] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.672390 19767 ts_tablet_manager.cc:1434] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.023s user 0.005s sys 0.003s
I20260317 16:17:53.673975 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-85f478d86a0949f381f0d88a657ba4e5 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:53.674434 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-85f478d86a0949f381f0d88a657ba4e5 on tablet 85f478d86a0949f381f0d88a657ba4e5 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:53.679864 19772 tablet_copy_client.cc:538] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:53.690258 19772 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:53.690845 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cf03ff443ed41d3bb50d992c1f34372 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:53.693150 19778 tablet_replica.cc:333] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:53.694736 19778 raft_consensus.cc:2243] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:53.695842 19778 raft_consensus.cc:2272] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:53.710889 19771 tablet_copy_client.cc:806] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:53.711550 19771 tablet_copy_client.cc:670] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:53.719050 19778 ts_tablet_manager.cc:1916] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:53.725307 19771 tablet_copy_client.cc:538] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:53.735572 19771 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:53.745529 19772 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:53.746518 19772 tablet_bootstrap.cc:492] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:53.747256 19772 ts_tablet_manager.cc:1403] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.057s user 0.029s sys 0.012s
I20260317 16:17:53.750375 19772 raft_consensus.cc:359] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.765858 19772 raft_consensus.cc:740] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:53.766582 19772 consensus_queue.cc:260] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.770793 19772 ts_tablet_manager.cc:1434] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260317 16:17:53.780857 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-8c9d82ef74c442299f4a3cf04a5bb2d7 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:53.781351 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-8c9d82ef74c442299f4a3cf04a5bb2d7 on tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:53.786115 19778 ts_tablet_manager.cc:1929] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260317 16:17:53.786742 19778 log.cc:1199] T 1cf03ff443ed41d3bb50d992c1f34372 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/1cf03ff443ed41d3bb50d992c1f34372
I20260317 16:17:53.788816 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 1cf03ff443ed41d3bb50d992c1f34372 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:53.797776 19772 ts_tablet_manager.cc:933] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:53.801625 19629 raft_consensus.cc:993] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:53.802145 19629 raft_consensus.cc:1081] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:53.804338 19746 raft_consensus.cc:993] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:53.804822 19746 raft_consensus.cc:1081] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:53.806156 18812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57326
I20260317 16:17:53.806847 18812 raft_consensus.cc:493] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:53.807278 18812 raft_consensus.cc:3060] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:53.811935 19771 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:53.812850 19771 tablet_bootstrap.cc:492] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:53.813510 19771 ts_tablet_manager.cc:1403] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.078s user 0.019s sys 0.010s
I20260317 16:17:53.816840 19771 raft_consensus.cc:359] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.817699 19771 raft_consensus.cc:740] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:53.818349 19771 consensus_queue.cc:260] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.825286 19771 ts_tablet_manager.cc:1434] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260317 16:17:53.827584 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-75ecbd9bfe9e4fe4b1f5146d5029d1c0 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:53.828121 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-75ecbd9bfe9e4fe4b1f5146d5029d1c0 on tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:53.827630 19772 tablet_copy_client.cc:323] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.194:35635
I20260317 16:17:53.830482 18137 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e390c786df0b48878244a08e889950a3"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47376
I20260317 16:17:53.831163 18137 raft_consensus.cc:493] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:53.831234 18222 tablet_copy_service.cc:140] P 88b3a81306894b71bf2eacc88c770eaa: Received BeginTabletCopySession request for tablet 77f02a05d0ee4e73bafe3055cfcf02f2 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:33718)
I20260317 16:17:53.831553 18137 raft_consensus.cc:3060] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:53.831755 18222 tablet_copy_service.cc:161] P 88b3a81306894b71bf2eacc88c770eaa: Beginning new tablet copy session on tablet 77f02a05d0ee4e73bafe3055cfcf02f2 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:33718: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-77f02a05d0ee4e73bafe3055cfcf02f2
I20260317 16:17:53.837988 18812 raft_consensus.cc:515] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.840690 18137 raft_consensus.cc:515] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.847918 18137 leader_election.cc:290] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:53.848290 18813 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:53.854934 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:53.856074 18286 raft_consensus.cc:3055] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:53.856381 18286 raft_consensus.cc:740] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:53.857061 18286 consensus_queue.cc:260] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.855005 18812 leader_election.cc:290] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:53.855261 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24063e0e6d6a4826899168d2524d5abc" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:53.849105 18813 raft_consensus.cc:3060] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:53.859457 18210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e390c786df0b48878244a08e889950a3" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:53.860005 18210 raft_consensus.cc:3055] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:53.860265 18210 raft_consensus.cc:740] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:53.860924 18210 consensus_queue.cc:260] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.862118 18210 raft_consensus.cc:3060] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:53.867962 18813 raft_consensus.cc:2468] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:53.869575 18222 tablet_copy_source_session.cc:215] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:53.871752 18100 leader_election.cc:304] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:53.858933 18208 raft_consensus.cc:3060] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:53.873646 19762 raft_consensus.cc:2804] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:53.874114 19762 raft_consensus.cc:697] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:53.874998 19762 consensus_queue.cc:237] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.879091 19772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f02a05d0ee4e73bafe3055cfcf02f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:53.891706 18210 raft_consensus.cc:2468] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 2.
I20260317 16:17:53.877130 18286 raft_consensus.cc:3060] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:53.882725 18208 raft_consensus.cc:2468] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:53.897677 18781 leader_election.cc:304] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:53.898732 19751 raft_consensus.cc:2804] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:53.899297 19751 raft_consensus.cc:697] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:53.900367 19751 consensus_queue.cc:237] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:53.902621 18286 raft_consensus.cc:2468] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:53.907804 18813 raft_consensus.cc:1217] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260317 16:17:53.909533 19772 tablet_copy_client.cc:806] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:53.910073 19772 tablet_copy_client.cc:670] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:53.915860 19772 tablet_copy_client.cc:538] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:53.927709 18032 catalog_manager.cc:5671] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 2 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:53.928121 18035 catalog_manager.cc:5671] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 021d78130690452ba094f4a392fb4378 (127.16.123.195) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:53.934563 19772 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:53.948987 19766 raft_consensus.cc:993] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:53.949528 19766 raft_consensus.cc:1081] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Signalling peer 94edf207909c4f8990bfe29ed1a87a6d to start an election
I20260317 16:17:53.951484 18134 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
from {username='slave'} at 127.0.0.1:47376
I20260317 16:17:53.952016 18134 raft_consensus.cc:493] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:53.952353 18134 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:53.964413 18134 raft_consensus.cc:515] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:53.965725 18134 leader_election.cc:290] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 3 election: Requested vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:53.967922 19772 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:53.968119 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:53.968689 18286 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:53.969189 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5389f9d0bc44e87a3f5897fb4b550ef" candidate_uuid: "94edf207909c4f8990bfe29ed1a87a6d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:53.969733 18208 raft_consensus.cc:3055] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:17:53.970022 18208 raft_consensus.cc:740] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:53.970703 18208 consensus_queue.cc:260] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:53.972218 18208 raft_consensus.cc:3060] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:53.968891 19772 tablet_bootstrap.cc:492] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:53.974998 19772 ts_tablet_manager.cc:1403] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.041s user 0.025s sys 0.007s
I20260317 16:17:53.978446 18286 raft_consensus.cc:2468] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 3.
I20260317 16:17:53.978133 19772 raft_consensus.cc:359] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.978866 18208 raft_consensus.cc:2468] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94edf207909c4f8990bfe29ed1a87a6d in term 3.
I20260317 16:17:53.979012 19772 raft_consensus.cc:740] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:53.979787 18103 leader_election.cc:304] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:53.980919 19758 raft_consensus.cc:2804] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:53.981442 19758 raft_consensus.cc:697] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Becoming Leader. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:53.982470 19758 consensus_queue.cc:237] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:53.979809 19772 consensus_queue.cc:260] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:53.992319 19772 ts_tablet_manager.cc:1434] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260317 16:17:53.998138 18222 tablet_copy_service.cc:342] P 88b3a81306894b71bf2eacc88c770eaa: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-77f02a05d0ee4e73bafe3055cfcf02f2 received from {username='slave'} at 127.0.0.1:33718
I20260317 16:17:53.998631 18222 tablet_copy_service.cc:434] P 88b3a81306894b71bf2eacc88c770eaa: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-77f02a05d0ee4e73bafe3055cfcf02f2 on tablet 77f02a05d0ee4e73bafe3055cfcf02f2 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:54.013003 18813 raft_consensus.cc:1217] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:54.031350 18035 catalog_manager.cc:5671] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: term changed from 2 to 3, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193). New cstate: current_term: 3 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:54.046132 18813 raft_consensus.cc:1217] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260317 16:17:54.328753 18812 raft_consensus.cc:1217] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20260317 16:17:54.341428 18812 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:54.342852 19758 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:54.357506 18204 raft_consensus.cc:1275] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:54.362700 19762 consensus_queue.cc:1048] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:54.400413 18208 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:54.402293 19718 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:54.422420 18281 raft_consensus.cc:1275] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:54.424011 19718 consensus_queue.cc:1048] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:54.430299 19782 raft_consensus.cc:1064] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:54.434168 18286 raft_consensus.cc:1275] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:54.435555 19762 consensus_queue.cc:1048] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:54.436770 19782 consensus_queue.cc:237] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:54.458417 18208 raft_consensus.cc:1275] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:17:54.462726 19762 consensus_queue.cc:1048] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:54.484865 19758 raft_consensus.cc:1064] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:54.496031 19758 consensus_queue.cc:237] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:54.544895 18210 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:54.545121 18137 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:54.546126 18812 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:54.547938 19595 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:54.549799 19595 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:54.549726 19752 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260317 16:17:54.563913 18286 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:54.565177 18813 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:54.574513 19782 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260317 16:17:54.582310 19809 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:54.608685 18204 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:54.611187 19809 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:54.611497 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:54.638007 19758 raft_consensus.cc:1064] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:54.663690 19752 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.669461 18813 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.669350 19805 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.673733 18133 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.690388 18285 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.701680 18817 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.710883 18035 catalog_manager.cc:5671] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:54.717782 18032 catalog_manager.cc:5671] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.723631 19766 raft_consensus.cc:1064] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:54.717186 19758 consensus_queue.cc:237] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:54.727886 19766 consensus_queue.cc:237] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:54.721199 18210 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.737125 18208 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.765029 18816 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:54.772025 18285 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:54.773100 19809 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:54.777920 18208 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 16:17:54.779781 19809 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:54.797439 19809 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:54.823766 18285 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:54.826066 19793 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:54.823769 18816 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:54.826679 18133 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:54.829578 19722 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:54.836057 19793 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:54.840705 19758 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.857089 18137 consensus_queue.cc:237] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:54.870404 18812 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:54.874959 19809 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:54.869498 18204 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.879797 18286 raft_consensus.cc:1275] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:54.881832 19808 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.888585 18817 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.893205 18813 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.898983 18812 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.896811 19766 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.903839 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:54.915496 19752 consensus_queue.cc:1048] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:54.930828 18286 raft_consensus.cc:2955] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.950428 18281 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.978899 18283 consensus_queue.cc:237] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:54.985797 18210 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:54.985037 18283 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.986306 18031 catalog_manager.cc:5671] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:54.991572 18812 raft_consensus.cc:1275] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:54.990878 19798 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260317 16:17:55.002751 18035 catalog_manager.cc:5671] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 2 to 3, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.004515 19805 consensus_queue.cc:1048] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:55.015235 19782 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.022086 18136 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77"
dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
mode: GRACEFUL
new_leader_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:33244
I20260317 16:17:55.022717 18136 raft_consensus.cc:606] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Received request to transfer leadership to TS 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:55.023716 18210 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.030938 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260317 16:17:55.034417 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:55.035223 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:55.047998 18813 raft_consensus.cc:2955] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.058142 18022 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 85f478d86a0949f381f0d88a657ba4e5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:55.071594 18033 catalog_manager.cc:5671] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 4 to 5, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:55.088629 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 85f478d86a0949f381f0d88a657ba4e5 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:55.103121 18032 catalog_manager.cc:5671] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 3 to 4, VOTER 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.108556 19827 tablet_replica.cc:333] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:55.112514 19827 raft_consensus.cc:2243] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:55.113408 19827 raft_consensus.cc:2272] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:55.114470 18136 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: true } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.137054 19827 ts_tablet_manager.cc:1916] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:55.152676 18136 consensus_queue.cc:237] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:55.158704 18198 tablet_service.cc:1558] Processing DeleteTablet for tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 with delete_type TABLET_DATA_TOMBSTONED (TS 88b3a81306894b71bf2eacc88c770eaa not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57484
I20260317 16:17:55.168354 18206 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:55.169137 19827 ts_tablet_manager.cc:1929] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260317 16:17:55.171422 19827 log.cc:1199] T 85f478d86a0949f381f0d88a657ba4e5 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/85f478d86a0949f381f0d88a657ba4e5
I20260317 16:17:55.171113 19808 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:55.173177 18204 consensus_queue.cc:237] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:55.176334 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 85f478d86a0949f381f0d88a657ba4e5 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:55.190148 19762 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.191164 18208 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.195012 19828 tablet_replica.cc:333] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:55.196437 19828 raft_consensus.cc:2243] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:55.197563 19828 raft_consensus.cc:2272] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:55.207646 18019 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:55.227645 19828 ts_tablet_manager.cc:1916] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:55.233243 18813 raft_consensus.cc:1275] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260317 16:17:55.234341 18282 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:55.238837 19808 consensus_queue.cc:1048] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:55.242058 19793 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260317 16:17:55.252399 18813 raft_consensus.cc:1275] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260317 16:17:55.254865 19828 ts_tablet_manager.cc:1929] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:55.255553 19828 log.cc:1199] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 88b3a81306894b71bf2eacc88c770eaa: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/wals/8c9d82ef74c442299f4a3cf04a5bb2d7
I20260317 16:17:55.257424 18282 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.258354 19793 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.257735 18817 raft_consensus.cc:2955] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.272260 18021 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 77f02a05d0ee4e73bafe3055cfcf02f2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260317 16:17:55.278194 19766 consensus_queue.cc:1048] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260317 16:17:55.288782 18032 catalog_manager.cc:5671] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index 3 to 4, VOTER 021d78130690452ba094f4a392fb4378 (127.16.123.195) evicted. New cstate: current_term: 1 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:55.291025 18021 catalog_manager.cc:5002] TS 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): tablet 8c9d82ef74c442299f4a3cf04a5bb2d7 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:55.297194 18813 raft_consensus.cc:2955] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.304939 18035 catalog_manager.cc:5671] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index 5 to 6, VOTER 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) evicted. New cstate: current_term: 3 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:55.308701 18268 tablet_service.cc:1558] Processing DeleteTablet for tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 with delete_type TABLET_DATA_TOMBSTONED (TS 021d78130690452ba094f4a392fb4378 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33376
I20260317 16:17:55.321075 18124 tablet_service.cc:1558] Processing DeleteTablet for tablet 77f02a05d0ee4e73bafe3055cfcf02f2 with delete_type TABLET_DATA_TOMBSTONED (TS 94edf207909c4f8990bfe29ed1a87a6d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47360
I20260317 16:17:55.322031 19827 tablet_replica.cc:333] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:55.320662 19829 tablet_replica.cc:333] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:55.323084 19827 raft_consensus.cc:2243] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:55.323956 19827 raft_consensus.cc:2272] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:55.325598 19829 raft_consensus.cc:2243] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:55.326730 19829 raft_consensus.cc:2272] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:55.328017 19827 ts_tablet_manager.cc:1916] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:55.331398 19829 ts_tablet_manager.cc:1916] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260317 16:17:55.349362 19827 ts_tablet_manager.cc:1929] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260317 16:17:55.349895 19827 log.cc:1199] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 94edf207909c4f8990bfe29ed1a87a6d: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/wals/77f02a05d0ee4e73bafe3055cfcf02f2
I20260317 16:17:55.350646 19829 ts_tablet_manager.cc:1929] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260317 16:17:55.351096 19829 log.cc:1199] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 021d78130690452ba094f4a392fb4378: Deleting WAL directory at /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/wals/75ecbd9bfe9e4fe4b1f5146d5029d1c0
I20260317 16:17:55.351342 18019 catalog_manager.cc:5002] TS 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): tablet 77f02a05d0ee4e73bafe3055cfcf02f2 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:55.352798 18022 catalog_manager.cc:5002] TS 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): tablet 75ecbd9bfe9e4fe4b1f5146d5029d1c0 (table test-workload [id=d8d01909e6e14a16a1638223e144843f]) successfully deleted
I20260317 16:17:55.507095 19752 raft_consensus.cc:993] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: : Instructing follower 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:55.507601 19758 raft_consensus.cc:1081] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Signalling peer 88b3a81306894b71bf2eacc88c770eaa to start an election
I20260317 16:17:55.509356 18204 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
from {username='slave'} at 127.0.0.1:57490
I20260317 16:17:55.509860 18204 raft_consensus.cc:493] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:55.510218 18204 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:55.533219 18204 raft_consensus.cc:515] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:55.534900 18204 leader_election.cc:290] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 4 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:55.535979 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "021d78130690452ba094f4a392fb4378"
I20260317 16:17:55.536706 18286 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:55.536439 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f162d501437442c1ad448749ff6b9d77" candidate_uuid: "88b3a81306894b71bf2eacc88c770eaa" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:55.537284 18136 raft_consensus.cc:3055] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:17:55.537588 18136 raft_consensus.cc:740] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94edf207909c4f8990bfe29ed1a87a6d, State: Running, Role: LEADER
I20260317 16:17:55.538166 18136 consensus_queue.cc:260] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:55.539175 18136 raft_consensus.cc:3060] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:17:55.550307 18286 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 4.
I20260317 16:17:55.553082 18178 leader_election.cc:304] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa; no voters:
I20260317 16:17:55.554450 19793 raft_consensus.cc:2804] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:17:55.555011 19793 raft_consensus.cc:697] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 4 LEADER]: Becoming Leader. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:55.557097 19793 consensus_queue.cc:237] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:55.572127 18136 raft_consensus.cc:2468] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3a81306894b71bf2eacc88c770eaa in term 4.
I20260317 16:17:55.574173 18032 catalog_manager.cc:5671] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: term changed from 3 to 4, leader changed from 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193) to 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194). New cstate: current_term: 4 leader_uuid: "88b3a81306894b71bf2eacc88c770eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:56.037899 18281 raft_consensus.cc:1275] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:17:56.039376 19766 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:56.069723 18136 raft_consensus.cc:1275] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Refusing update from remote peer 88b3a81306894b71bf2eacc88c770eaa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260317 16:17:56.071556 19766 consensus_queue.cc:1048] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260317 16:17:56.115408 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 46935.8us
I20260317 16:17:56.132077 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:56.246199 18208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0"
dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
mode: GRACEFUL
new_leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:46506
I20260317 16:17:56.246708 18208 raft_consensus.cc:606] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Received request to transfer leadership to TS 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:56.248296 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260317 16:17:56.249907 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:56.250752 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:56.397557 18286 consensus_queue.cc:237] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
W20260317 16:17:56.405547 18249 consensus_peers.cc:597] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7aafcd49c9d54f548b25a51935f79523. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:56.405856 18204 raft_consensus.cc:1275] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:56.406090 18137 raft_consensus.cc:1275] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:56.407683 19782 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:56.408733 19835 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:17:56.430624 19782 raft_consensus.cc:2955] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.434015 18137 raft_consensus.cc:2955] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.433295 18204 raft_consensus.cc:2955] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.480413 19793 raft_consensus.cc:993] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: : Instructing follower 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:56.484012 19722 raft_consensus.cc:1081] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Signalling peer 381cc8622e1f4a53a7ff2b23ed2bcd38 to start an election
I20260317 16:17:56.489956 18813 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0"
dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
from {username='slave'} at 127.0.0.1:57316
I20260317 16:17:56.490562 18813 raft_consensus.cc:493] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260317 16:17:56.491084 18813 raft_consensus.cc:3060] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:56.490099 18137 consensus_queue.cc:237] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:56.497642 18032 catalog_manager.cc:5671] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:56.512312 18813 raft_consensus.cc:515] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:56.525067 18813 leader_election.cc:290] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:56.526909 18286 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:56.527666 18136 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94edf207909c4f8990bfe29ed1a87a6d"
I20260317 16:17:56.528249 18136 raft_consensus.cc:3060] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:17:56.527066 18100 consensus_peers.cc:597] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ee9125e28d14f3c8f458dda86c4ee15. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:56.530484 18204 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260317 16:17:56.531862 18208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdb770e745e0447faf1334e5d343ace0" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "88b3a81306894b71bf2eacc88c770eaa"
I20260317 16:17:56.532418 18208 raft_consensus.cc:3055] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:17:56.533465 18208 raft_consensus.cc:740] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b3a81306894b71bf2eacc88c770eaa, State: Running, Role: LEADER
I20260317 16:17:56.540545 19831 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:56.543058 19762 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:56.544203 18208 consensus_queue.cc:260] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:56.550338 18208 raft_consensus.cc:3060] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:56.567210 18136 raft_consensus.cc:2468] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:56.569890 18779 leader_election.cc:304] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d; no voters:
I20260317 16:17:56.571002 18208 raft_consensus.cc:2468] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381cc8622e1f4a53a7ff2b23ed2bcd38 in term 2.
I20260317 16:17:56.573923 19838 raft_consensus.cc:2804] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:56.575948 19838 raft_consensus.cc:697] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Becoming Leader. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Running, Role: LEADER
I20260317 16:17:56.578828 19831 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.579128 18206 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.577745 19838 consensus_queue.cc:237] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:56.610980 18281 consensus_queue.cc:237] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:56.616863 18031 catalog_manager.cc:5671] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: term changed from 1 to 2, leader changed from 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194) to 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260317 16:17:56.623700 18206 raft_consensus.cc:1275] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260317 16:17:56.623113 18286 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.626150 19835 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:56.629169 18133 raft_consensus.cc:1275] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260317 16:17:56.633811 18249 consensus_peers.cc:597] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 -> Peer 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243): Couldn't send request to peer 381cc8622e1f4a53a7ff2b23ed2bcd38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 702896c0cf9948d68e4c4148743bc5c8. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:56.634039 19843 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:17:56.675347 19844 raft_consensus.cc:2955] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.687552 18133 raft_consensus.cc:2955] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.737491 18033 catalog_manager.cc:5671] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d reported cstate change: config changed from index -1 to 4, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 3 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260317 16:17:56.818953 18206 raft_consensus.cc:2955] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } } }
I20260317 16:17:56.829279 18033 catalog_manager.cc:5671] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 reported cstate change: config changed from index -1 to 3, NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) added. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260317 16:17:56.890125 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 55241us
I20260317 16:17:56.914976 19857 ts_tablet_manager.cc:933] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:56.933526 19857 tablet_copy_client.cc:323] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:56.944253 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 7aafcd49c9d54f548b25a51935f79523 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:56.949023 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 7aafcd49c9d54f548b25a51935f79523 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-7aafcd49c9d54f548b25a51935f79523
I20260317 16:17:56.978266 18295 tablet_copy_source_session.cc:215] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:56.988196 19857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aafcd49c9d54f548b25a51935f79523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:57.014969 19857 tablet_copy_client.cc:806] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:57.015617 19857 tablet_copy_client.cc:670] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:57.022212 19857 tablet_copy_client.cc:538] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:57.024032 19860 ts_tablet_manager.cc:933] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673)
I20260317 16:17:57.036620 19857 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:57.060282 19860 tablet_copy_client.cc:323] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.193:45673
I20260317 16:17:57.064080 18147 tablet_copy_service.cc:140] P 94edf207909c4f8990bfe29ed1a87a6d: Received BeginTabletCopySession request for tablet 4ee9125e28d14f3c8f458dda86c4ee15 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:44608)
I20260317 16:17:57.066272 18147 tablet_copy_service.cc:161] P 94edf207909c4f8990bfe29ed1a87a6d: Beginning new tablet copy session on tablet 4ee9125e28d14f3c8f458dda86c4ee15 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:44608: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-4ee9125e28d14f3c8f458dda86c4ee15
I20260317 16:17:57.069268 18204 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:57.074872 19796 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:57.097918 18147 tablet_copy_source_session.cc:215] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:57.101296 19860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9125e28d14f3c8f458dda86c4ee15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:57.102644 19857 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:57.103878 19857 tablet_bootstrap.cc:492] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:57.104609 19857 ts_tablet_manager.cc:1403] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.003s
I20260317 16:17:57.107751 19857 raft_consensus.cc:359] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:57.109114 19857 raft_consensus.cc:740] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:57.111289 19857 consensus_queue.cc:260] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:57.117995 19865 ts_tablet_manager.cc:933] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Initiating tablet copy from peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:57.119422 19857 ts_tablet_manager.cc:1434] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260317 16:17:57.122519 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-7aafcd49c9d54f548b25a51935f79523 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:57.123000 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-7aafcd49c9d54f548b25a51935f79523 on tablet 7aafcd49c9d54f548b25a51935f79523 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:57.125605 18133 raft_consensus.cc:1275] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Refusing update from remote peer 381cc8622e1f4a53a7ff2b23ed2bcd38: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260317 16:17:57.129954 19859 consensus_queue.cc:1048] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:57.131054 19860 tablet_copy_client.cc:806] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:57.131650 19860 tablet_copy_client.cc:670] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:57.153967 19860 tablet_copy_client.cc:538] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:57.231645 19860 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:57.239310 19865 tablet_copy_client.cc:323] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.123.195:44851
I20260317 16:17:57.251592 18295 tablet_copy_service.cc:140] P 021d78130690452ba094f4a392fb4378: Received BeginTabletCopySession request for tablet 702896c0cf9948d68e4c4148743bc5c8 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 ({username='slave'} at 127.0.0.1:45904)
I20260317 16:17:57.252156 18295 tablet_copy_service.cc:161] P 021d78130690452ba094f4a392fb4378: Beginning new tablet copy session on tablet 702896c0cf9948d68e4c4148743bc5c8 from peer 381cc8622e1f4a53a7ff2b23ed2bcd38 at {username='slave'} at 127.0.0.1:45904: session id = 381cc8622e1f4a53a7ff2b23ed2bcd38-702896c0cf9948d68e4c4148743bc5c8
I20260317 16:17:57.271543 18295 tablet_copy_source_session.cc:215] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: Tablet Copy: opened 0 blocks and 1 log segments
I20260317 16:17:57.277786 19865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702896c0cf9948d68e4c4148743bc5c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:17:57.303412 19865 tablet_copy_client.cc:806] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 0 data blocks...
I20260317 16:17:57.306452 19865 tablet_copy_client.cc:670] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Starting download of 1 WAL segments...
I20260317 16:17:57.311482 19865 tablet_copy_client.cc:538] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260317 16:17:57.316486 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:57.317008 19860 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:57.318187 19860 tablet_bootstrap.cc:492] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:57.318993 19860 ts_tablet_manager.cc:1403] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.093s user 0.026s sys 0.011s
I20260317 16:17:57.322355 19860 raft_consensus.cc:359] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:57.323364 19860 raft_consensus.cc:740] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:57.324152 19860 consensus_queue.cc:260] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:57.332871 19865 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap starting.
I20260317 16:17:57.340737 19860 ts_tablet_manager.cc:1434] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.021s user 0.005s sys 0.003s
I20260317 16:17:57.352588 18147 tablet_copy_service.cc:342] P 94edf207909c4f8990bfe29ed1a87a6d: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-4ee9125e28d14f3c8f458dda86c4ee15 received from {username='slave'} at 127.0.0.1:44608
I20260317 16:17:57.353170 18147 tablet_copy_service.cc:434] P 94edf207909c4f8990bfe29ed1a87a6d: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-4ee9125e28d14f3c8f458dda86c4ee15 on tablet 4ee9125e28d14f3c8f458dda86c4ee15 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:57.377395 19865 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 16:17:57.379611 19865 tablet_bootstrap.cc:492] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Bootstrap complete.
I20260317 16:17:57.381748 19865 ts_tablet_manager.cc:1403] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent bootstrapping tablet: real 0.049s user 0.032s sys 0.007s
I20260317 16:17:57.384521 19865 raft_consensus.cc:359] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:57.385326 19865 raft_consensus.cc:740] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381cc8622e1f4a53a7ff2b23ed2bcd38, State: Initialized, Role: LEARNER
I20260317 16:17:57.387082 19865 consensus_queue.cc:260] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: NON_VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: true } }
I20260317 16:17:57.389788 19865 ts_tablet_manager.cc:1434] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20260317 16:17:57.394348 18295 tablet_copy_service.cc:342] P 021d78130690452ba094f4a392fb4378: Request end of tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-702896c0cf9948d68e4c4148743bc5c8 received from {username='slave'} at 127.0.0.1:45904
I20260317 16:17:57.394961 18295 tablet_copy_service.cc:434] P 021d78130690452ba094f4a392fb4378: ending tablet copy session 381cc8622e1f4a53a7ff2b23ed2bcd38-702896c0cf9948d68e4c4148743bc5c8 on tablet 702896c0cf9948d68e4c4148743bc5c8 with peer 381cc8622e1f4a53a7ff2b23ed2bcd38
I20260317 16:17:57.403972 16879 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260317 16:17:57.405398 16879 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260317 16:17:57.407011 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:57.417075 18813 raft_consensus.cc:1217] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260317 16:17:57.481602 16879 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260317 16:17:57.552431 16879 tablet_server.cc:178] TabletServer@127.16.123.193:0 shutting down...
W20260317 16:17:57.556105 18175 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20260317 16:17:57.572857 18813 raft_consensus.cc:1217] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260317 16:17:57.596695 18817 raft_consensus.cc:1217] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
W20260317 16:17:57.603034 18083 auto_rebalancer.cc:798] Removing replace marker failed: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:57.603927 18083 auto_rebalancer.cc:806] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:57.604259 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:57.616156 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 60086.6us
W20260317 16:17:57.784674 18779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 44331.9us
W20260317 16:17:57.795974 18175 consensus_peers.cc:597] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:57.835729 18782 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:44851, user_credentials={real_user=slave}} blocked reactor thread for 141274us
W20260317 16:17:57.845479 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 90307.4us
W20260317 16:17:57.885300 18175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 75236us
W20260317 16:17:57.894752 18251 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35635, user_credentials={real_user=slave}} blocked reactor thread for 72509.6us
W20260317 16:17:57.910941 18779 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:45673, user_credentials={real_user=slave}} blocked reactor thread for 80413.8us
W20260317 16:17:57.919659 18779 consensus_peers.cc:597] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:57.921087 18779 consensus_peers.cc:597] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:57.928037 18175 consensus_peers.cc:597] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:57.933815 18175 consensus_peers.cc:597] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:57.967252 18249 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:40243, user_credentials={real_user=slave}} blocked reactor thread for 106454us
W20260317 16:17:57.991822 18779 consensus_peers.cc:597] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:57.999938 18779 consensus_peers.cc:597] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.001047 18779 consensus_peers.cc:597] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.003978 18779 consensus_peers.cc:597] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.006258 18779 consensus_peers.cc:597] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.000371 18175 consensus_peers.cc:597] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.032531 18779 consensus_peers.cc:597] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.031728 18249 consensus_peers.cc:597] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.033987 18779 consensus_peers.cc:597] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.034132 18249 consensus_peers.cc:597] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.034875 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:17:58.041023 16879 tablet_replica.cc:333] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.042321 16879 raft_consensus.cc:2243] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Raft consensus shutting down.
W20260317 16:17:58.043084 18779 consensus_peers.cc:597] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.043851 16879 raft_consensus.cc:2272] T c5389f9d0bc44e87a3f5897fb4b550ef P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.047971 16879 tablet_replica.cc:333] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
W20260317 16:17:58.049387 18175 consensus_peers.cc:597] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.049997 18175 consensus_peers.cc:597] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.051275 16879 raft_consensus.cc:2243] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.052942 16879 raft_consensus.cc:2272] T bfaeab6c75f94e128924d831ee8ef55b P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.054670 18175 consensus_peers.cc:597] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.057017 16879 tablet_replica.cc:333] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.057976 16879 raft_consensus.cc:2243] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.058997 16879 raft_consensus.cc:2272] T 0eb8a46e02034bd199afdbe545379351 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.062183 16879 tablet_replica.cc:333] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.063082 16879 raft_consensus.cc:2243] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.064172 16879 raft_consensus.cc:2272] T e2a036705e4a4da5a5392d119edb244e P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.067153 16879 tablet_replica.cc:333] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.068009 16879 raft_consensus.cc:2243] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.068522 16879 raft_consensus.cc:2272] T f1e740c77f5f4cb6916938c76fd04455 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.072125 16879 tablet_replica.cc:333] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.073015 16879 raft_consensus.cc:2243] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.074169 16879 raft_consensus.cc:2272] T 8d3017628c5244c3a8bb091d0a4768fd P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.078586 16879 tablet_replica.cc:333] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.079483 16879 raft_consensus.cc:2243] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.080046 16879 raft_consensus.cc:2272] T 9545c351b28449c2a44da8b9a94c7f11 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.080106 18249 consensus_peers.cc:597] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.080717 18249 consensus_peers.cc:597] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.081283 18249 consensus_peers.cc:597] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.081856 18249 consensus_peers.cc:597] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.082542 18175 consensus_peers.cc:597] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.082374 18249 consensus_peers.cc:597] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.083115 16879 tablet_replica.cc:333] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.084625 16879 raft_consensus.cc:2243] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.085423 16879 raft_consensus.cc:2272] T 9e145759d6de4334931c0039a97abdd8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.086328 19835 raft_consensus.cc:1064] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:58.088383 16879 tablet_replica.cc:333] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.089337 16879 raft_consensus.cc:2243] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.089130 19835 consensus_queue.cc:237] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:58.090284 16879 raft_consensus.cc:2272] T e390c786df0b48878244a08e889950a3 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.093127 16879 tablet_replica.cc:333] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.094987 16879 raft_consensus.cc:2243] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.095489 16879 raft_consensus.cc:2272] T f162d501437442c1ad448749ff6b9d77 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.096732 18249 consensus_peers.cc:597] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.096760 18175 consensus_peers.cc:597] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.097090 19880 raft_consensus.cc:1064] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:58.100191 16879 tablet_replica.cc:333] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.099844 19880 consensus_queue.cc:237] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:58.101334 16879 raft_consensus.cc:2243] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.102111 16879 raft_consensus.cc:2272] T 9246439ff23646f8abdf7bba7f3a9172 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.108696 16879 tablet_replica.cc:333] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.110591 16879 raft_consensus.cc:2243] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.111609 16879 raft_consensus.cc:2272] T 96d59cc0e53747428c5ff7f5178f11ca P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.114127 18813 raft_consensus.cc:1275] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:58.115933 16879 tablet_replica.cc:333] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.116106 19835 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:58.116853 16879 raft_consensus.cc:2243] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
W20260317 16:17:58.117218 18249 consensus_peers.cc:597] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.118105 16879 raft_consensus.cc:2272] T 70f2fc6f0fd640fcb531a01d7881a614 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.120977 16879 tablet_replica.cc:333] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.121891 16879 raft_consensus.cc:2243] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.122501 16879 raft_consensus.cc:2272] T 8284701fdaf640daa689a2534914514c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.124923 18208 raft_consensus.cc:1275] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:58.126428 16879 tablet_replica.cc:333] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.127342 16879 raft_consensus.cc:2243] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.128000 16879 raft_consensus.cc:2272] T 7aafcd49c9d54f548b25a51935f79523 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.130081 19835 consensus_queue.cc:1048] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260317 16:17:58.130661 18204 raft_consensus.cc:1275] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:58.130873 16879 tablet_replica.cc:333] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.131791 16879 raft_consensus.cc:2243] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.132284 16879 raft_consensus.cc:2272] T 47801c6cdf6f461181a89246716390d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.133167 19891 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:58.135048 16879 tablet_replica.cc:333] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.135597 18813 raft_consensus.cc:1275] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEARNER]: Refusing update from remote peer 021d78130690452ba094f4a392fb4378: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260317 16:17:58.135818 16879 raft_consensus.cc:2243] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.136294 16879 raft_consensus.cc:2272] T d88548519a0f473680897b8a7fc1e9b6 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.139362 16879 tablet_replica.cc:333] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.140296 16879 raft_consensus.cc:2243] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.140352 19891 consensus_queue.cc:1048] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:17:58.142046 16879 raft_consensus.cc:2272] T 44eefa978130466ead1b5828a0e0c4f3 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.151013 16879 tablet_replica.cc:333] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.153616 16879 raft_consensus.cc:2243] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.154805 16879 raft_consensus.cc:2272] T 94f2b31d108e461d932c2477e0f78fd0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.166361 16879 tablet_replica.cc:333] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.167779 16879 raft_consensus.cc:2243] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.168766 16879 raft_consensus.cc:2272] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.170858 18249 consensus_peers.cc:597] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.171468 16879 tablet_replica.cc:333] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.172287 16879 raft_consensus.cc:2243] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.172842 19884 raft_consensus.cc:1064] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: attempting to promote NON_VOTER 381cc8622e1f4a53a7ff2b23ed2bcd38 to VOTER
I20260317 16:17:58.175513 19884 consensus_queue.cc:237] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:58.178169 16879 raft_consensus.cc:2272] T fdb770e745e0447faf1334e5d343ace0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.181120 18779 consensus_peers.cc:597] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.182477 16879 tablet_replica.cc:333] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.183863 16879 raft_consensus.cc:2243] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.184419 16879 raft_consensus.cc:2272] T 17fe426def0d4f378a44ce5ec328f2e0 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.187525 16879 tablet_replica.cc:333] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.188400 16879 raft_consensus.cc:2243] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.187722 18210 raft_consensus.cc:2955] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.189428 16879 raft_consensus.cc:2272] T 07a6f1b891ac4c3492ae2548ff2fc357 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.192225 16879 tablet_replica.cc:333] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.193169 16879 raft_consensus.cc:2243] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.193737 16879 raft_consensus.cc:2272] T 702896c0cf9948d68e4c4148743bc5c8 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.197284 16879 tablet_replica.cc:333] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.197947 16879 raft_consensus.cc:2243] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.198402 16879 raft_consensus.cc:2272] T fdf3aa784b554e5386dcc11e2550e2f5 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.201660 16879 tablet_replica.cc:333] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.184763 19880 raft_consensus.cc:2955] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.203220 16879 raft_consensus.cc:2243] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.203850 16879 raft_consensus.cc:2272] T ac85b776838146d9904478a4ef0ae824 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.206704 16879 tablet_replica.cc:333] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.207659 16879 raft_consensus.cc:2243] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.208405 16879 raft_consensus.cc:2272] T 9844da16b03545c5a9e5534b29b4343c P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.211289 16879 tablet_replica.cc:333] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.212154 16879 raft_consensus.cc:2243] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.213125 16879 raft_consensus.cc:2272] T 5ded1f55ede142cbbde7c675a0f72d15 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.215895 16879 tablet_replica.cc:333] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.217283 16879 raft_consensus.cc:2243] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.218315 16879 raft_consensus.cc:2272] T 701d2829bc6a450a94470d7f852ecdc8 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.220664 18208 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:58.222355 19831 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:58.221388 16879 tablet_replica.cc:333] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.224176 18286 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:58.226707 19882 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260317 16:17:58.227237 18813 raft_consensus.cc:1275] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEARNER]: Refusing update from remote peer 94edf207909c4f8990bfe29ed1a87a6d: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260317 16:17:58.228636 19831 consensus_queue.cc:1048] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 16:17:58.230172 16879 raft_consensus.cc:2243] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.231309 16879 raft_consensus.cc:2272] T 87a9551ad4c24868805c90f4fa899ff0 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.233505 19890 raft_consensus.cc:2955] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.236707 18813 raft_consensus.cc:2955] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.239089 16879 tablet_replica.cc:333] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.240357 16879 raft_consensus.cc:2243] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.241402 16879 raft_consensus.cc:2272] T 9f14079a4ea34cbda92828ca274134f4 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.242105 18812 raft_consensus.cc:2955] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.245364 16879 tablet_replica.cc:333] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.246317 16879 raft_consensus.cc:2243] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.252499 16879 raft_consensus.cc:2272] T 4a773cd94be84692a95e55317c98a95c P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.257377 16879 tablet_replica.cc:333] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.258352 16879 raft_consensus.cc:2243] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.259200 16879 raft_consensus.cc:2272] T c85ea9795a2c43a38f1a9c3f3741db36 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.260402 19882 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.264791 18817 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.265017 18286 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.269625 18035 catalog_manager.cc:5671] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.278479 16879 tablet_replica.cc:333] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.279426 16879 raft_consensus.cc:2243] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.280406 16879 raft_consensus.cc:2272] T fb31cad8b88240acb9c101027ea444c1 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.285854 18210 raft_consensus.cc:2955] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.285957 18206 raft_consensus.cc:2955] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.296312 16879 tablet_replica.cc:333] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.297379 16879 raft_consensus.cc:2243] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.298468 16879 raft_consensus.cc:2272] T f77a63cdb5604ab493b7f5f9dc921763 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.301690 16879 tablet_replica.cc:333] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.304162 16879 raft_consensus.cc:2243] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.304948 16879 raft_consensus.cc:2272] T dbff882fd6aa4aa0a32d72d1674abc76 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.310464 16879 tablet_replica.cc:333] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.311235 16879 raft_consensus.cc:2243] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.312036 16879 raft_consensus.cc:2272] T 6b643cd863d14f36a3b9341312ef4770 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.323208 16879 tablet_replica.cc:333] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.324156 16879 raft_consensus.cc:2243] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.325350 16879 raft_consensus.cc:2272] T 4ee9125e28d14f3c8f458dda86c4ee15 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.329368 18033 catalog_manager.cc:5671] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa reported cstate change: config changed from index 3 to 4, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "021d78130690452ba094f4a392fb4378" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } attrs { replace: true } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.332414 16879 tablet_replica.cc:333] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.338157 16879 raft_consensus.cc:2243] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.338739 16879 raft_consensus.cc:2272] T 09f47966c253483393878fddf97ac206 P 94edf207909c4f8990bfe29ed1a87a6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.342489 16879 tablet_replica.cc:333] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.343237 16879 raft_consensus.cc:2243] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.344012 16879 raft_consensus.cc:2272] T 55ecf327ed214b02bef87b8c05d54c26 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.348104 16879 tablet_replica.cc:333] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.348843 16879 raft_consensus.cc:2243] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.349906 16879 raft_consensus.cc:2272] T fe3f3aabd9b444868c9732a8e5fa7440 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.352473 16879 tablet_replica.cc:333] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.353139 16879 raft_consensus.cc:2243] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.367221 16879 raft_consensus.cc:2272] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 94edf207909c4f8990bfe29ed1a87a6d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.370162 16879 tablet_replica.cc:333] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.370507 18032 catalog_manager.cc:5671] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 reported cstate change: config changed from index 4 to 5, 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "94edf207909c4f8990bfe29ed1a87a6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } } }
I20260317 16:17:58.371757 16879 raft_consensus.cc:2243] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.372766 16879 raft_consensus.cc:2272] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 94edf207909c4f8990bfe29ed1a87a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.375131 16879 tablet_replica.cc:333] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d: stopping tablet replica
I20260317 16:17:58.375792 16879 raft_consensus.cc:2243] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.376333 16879 raft_consensus.cc:2272] T 2f120096f51b4b3a992048069e2ae182 P 94edf207909c4f8990bfe29ed1a87a6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.549618 16879 tablet_server.cc:195] TabletServer@127.16.123.193:0 shutdown complete.
I20260317 16:17:58.707173 16879 tablet_server.cc:178] TabletServer@127.16.123.194:0 shutting down...
W20260317 16:17:58.748544 18251 consensus_peers.cc:597] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.752089 18251 consensus_peers.cc:597] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.759057 18251 consensus_peers.cc:597] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:58.779062 18251 consensus_peers.cc:597] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.785512 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:17:58.786840 16879 tablet_replica.cc:333] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.787880 16879 raft_consensus.cc:2243] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.788570 16879 raft_consensus.cc:2272] T c5389f9d0bc44e87a3f5897fb4b550ef P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.790958 16879 tablet_replica.cc:333] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.791669 16879 raft_consensus.cc:2243] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.793340 16879 raft_consensus.cc:2272] T a8ed2db954354e669de7909d315142aa P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.798076 16879 tablet_replica.cc:333] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.799135 16879 raft_consensus.cc:2243] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.800841 16879 raft_consensus.cc:2272] T e2a036705e4a4da5a5392d119edb244e P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.804025 16879 tablet_replica.cc:333] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.805198 16879 raft_consensus.cc:2243] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.806509 16879 raft_consensus.cc:2272] T b451ac456d424629ae5ce6040cb15bd2 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.811564 16879 tablet_replica.cc:333] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.813071 16879 raft_consensus.cc:2243] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.814172 16879 raft_consensus.cc:2272] T f1e740c77f5f4cb6916938c76fd04455 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.817463 16879 tablet_replica.cc:333] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.818226 16879 raft_consensus.cc:2243] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.819543 16879 raft_consensus.cc:2272] T 8d3017628c5244c3a8bb091d0a4768fd P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.826453 16879 tablet_replica.cc:333] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.827396 16879 raft_consensus.cc:2243] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.829319 16879 raft_consensus.cc:2272] T f77a63cdb5604ab493b7f5f9dc921763 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.830185 18251 consensus_peers.cc:597] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.831836 16879 tablet_replica.cc:333] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.832472 16879 raft_consensus.cc:2243] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.833151 16879 raft_consensus.cc:2272] T 1897e8c1602b4aeda93d9b8207b3ac45 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.836675 16879 tablet_replica.cc:333] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.837513 16879 raft_consensus.cc:2243] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.839087 16879 raft_consensus.cc:2272] T f162d501437442c1ad448749ff6b9d77 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.844478 16879 tablet_replica.cc:333] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.845255 16879 raft_consensus.cc:2243] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.845860 16879 raft_consensus.cc:2272] T 9545c351b28449c2a44da8b9a94c7f11 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.850800 16879 tablet_replica.cc:333] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
W20260317 16:17:58.851168 18251 consensus_peers.cc:597] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.851465 16879 raft_consensus.cc:2243] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.852423 16879 raft_consensus.cc:2272] T e390c786df0b48878244a08e889950a3 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.855356 16879 tablet_replica.cc:333] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.856545 16879 raft_consensus.cc:2243] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.857569 16879 raft_consensus.cc:2272] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.861758 16879 tablet_replica.cc:333] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.862818 16879 raft_consensus.cc:2243] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.864246 16879 raft_consensus.cc:2272] T fdf3aa784b554e5386dcc11e2550e2f5 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.864555 18251 consensus_peers.cc:597] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.867208 16879 tablet_replica.cc:333] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.868115 16879 raft_consensus.cc:2243] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.869221 16879 raft_consensus.cc:2272] T 9246439ff23646f8abdf7bba7f3a9172 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.871748 16879 tablet_replica.cc:333] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.872504 16879 raft_consensus.cc:2243] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.873116 16879 raft_consensus.cc:2272] T 898420a8ccf84bbdba91710ca9c08ef1 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.875474 16879 tablet_replica.cc:333] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.876689 16879 raft_consensus.cc:2243] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.877348 16879 raft_consensus.cc:2272] T 24063e0e6d6a4826899168d2524d5abc P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.879639 16879 tablet_replica.cc:333] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.880287 16879 raft_consensus.cc:2243] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.881793 16879 raft_consensus.cc:2272] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.884611 16879 tablet_replica.cc:333] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.885434 16879 raft_consensus.cc:2243] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.885864 16879 raft_consensus.cc:2272] T 44eefa978130466ead1b5828a0e0c4f3 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.888516 16879 tablet_replica.cc:333] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.889256 16879 raft_consensus.cc:2243] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.889816 16879 raft_consensus.cc:2272] T 5ded1f55ede142cbbde7c675a0f72d15 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.892496 16879 tablet_replica.cc:333] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.894443 16879 raft_consensus.cc:2243] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.895090 16879 raft_consensus.cc:2272] T 96d59cc0e53747428c5ff7f5178f11ca P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.897485 16879 tablet_replica.cc:333] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.898252 16879 raft_consensus.cc:2243] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.898923 16879 raft_consensus.cc:2272] T 70f2fc6f0fd640fcb531a01d7881a614 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.901953 16879 tablet_replica.cc:333] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.902786 16879 raft_consensus.cc:2243] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.903738 16879 raft_consensus.cc:2272] T d88548519a0f473680897b8a7fc1e9b6 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.908083 16879 tablet_replica.cc:333] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.909152 16879 raft_consensus.cc:2243] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.909912 16879 raft_consensus.cc:2272] T 8284701fdaf640daa689a2534914514c P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.912703 16879 tablet_replica.cc:333] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.913601 16879 raft_consensus.cc:2243] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.914427 16879 raft_consensus.cc:2272] T 7aafcd49c9d54f548b25a51935f79523 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.915184 18781 consensus_peers.cc:597] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.918108 16879 tablet_replica.cc:333] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.919332 16879 raft_consensus.cc:2243] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.920377 16879 raft_consensus.cc:2272] T 47801c6cdf6f461181a89246716390d7 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.923110 16879 tablet_replica.cc:333] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.923861 16879 raft_consensus.cc:2243] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.924917 16879 raft_consensus.cc:2272] T 519bc447dbc1455c844945ba63c96914 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.927805 16879 tablet_replica.cc:333] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.928582 16879 raft_consensus.cc:2243] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.929095 16879 raft_consensus.cc:2272] T fdb770e745e0447faf1334e5d343ace0 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.932047 16879 tablet_replica.cc:333] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.932920 16879 raft_consensus.cc:2243] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.933696 16879 raft_consensus.cc:2272] T 702896c0cf9948d68e4c4148743bc5c8 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.939318 16879 tablet_replica.cc:333] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.940056 16879 raft_consensus.cc:2243] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.940625 16879 raft_consensus.cc:2272] T 0b2a4d319ba9480b9c950e12b7329101 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.946710 16879 tablet_replica.cc:333] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.948211 16879 raft_consensus.cc:2243] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.949154 16879 raft_consensus.cc:2272] T 4ee9125e28d14f3c8f458dda86c4ee15 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.950086 18781 consensus_peers.cc:597] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.952327 16879 tablet_replica.cc:333] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.953100 16879 raft_consensus.cc:2243] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.953644 16879 raft_consensus.cc:2272] T 09f47966c253483393878fddf97ac206 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.956507 16879 tablet_replica.cc:333] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.957751 16879 raft_consensus.cc:2243] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.959218 16879 raft_consensus.cc:2272] T 07c5d59b5b404eec925d5c4681953edb P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.961336 18781 consensus_peers.cc:597] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.963232 16879 tablet_replica.cc:333] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.965210 16879 raft_consensus.cc:2243] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.966243 16879 raft_consensus.cc:2272] T 1cf03ff443ed41d3bb50d992c1f34372 P 88b3a81306894b71bf2eacc88c770eaa [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.967684 18781 consensus_peers.cc:597] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.969964 16879 tablet_replica.cc:333] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.970736 16879 raft_consensus.cc:2243] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.971746 16879 raft_consensus.cc:2272] T c94382dceadb48228dd3467bf673a11a P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.974849 16879 tablet_replica.cc:333] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.975570 16879 raft_consensus.cc:2243] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.976459 16879 raft_consensus.cc:2272] T 84ded5b9676242a18139f88dc63af2b4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.976927 18251 consensus_peers.cc:597] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.978914 16879 tablet_replica.cc:333] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.979521 16879 raft_consensus.cc:2243] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.980096 16879 raft_consensus.cc:2272] T 85f478d86a0949f381f0d88a657ba4e5 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.982240 16879 tablet_replica.cc:333] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.982955 16879 raft_consensus.cc:2243] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.983973 16879 raft_consensus.cc:2272] T 4a773cd94be84692a95e55317c98a95c P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.986392 16879 tablet_replica.cc:333] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.987366 16879 raft_consensus.cc:2243] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.988076 16879 raft_consensus.cc:2272] T e6aca2493a994ed888cd663ddcf97582 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:58.989280 18781 consensus_peers.cc:597] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:58.990382 16879 tablet_replica.cc:333] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.991330 16879 raft_consensus.cc:2243] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:58.993273 16879 raft_consensus.cc:2272] T dbff882fd6aa4aa0a32d72d1674abc76 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:58.996461 16879 tablet_replica.cc:333] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:58.997169 16879 raft_consensus.cc:2243] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:58.997761 16879 raft_consensus.cc:2272] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 88b3a81306894b71bf2eacc88c770eaa [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.000993 16879 tablet_replica.cc:333] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:59.001979 16879 raft_consensus.cc:2243] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.004161 16879 raft_consensus.cc:2272] T ab7a4647163b461f87a0a249491e566c P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.008307 16879 tablet_replica.cc:333] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:59.009065 16879 raft_consensus.cc:2243] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.009727 16879 raft_consensus.cc:2272] T 2f120096f51b4b3a992048069e2ae182 P 88b3a81306894b71bf2eacc88c770eaa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.012383 16879 tablet_replica.cc:333] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:59.012989 16879 raft_consensus.cc:2243] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.013448 16879 raft_consensus.cc:2272] T 9f14079a4ea34cbda92828ca274134f4 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.016099 16879 tablet_replica.cc:333] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:59.017055 16879 raft_consensus.cc:2243] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.020845 16879 raft_consensus.cc:2272] T 87a9551ad4c24868805c90f4fa899ff0 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.021660 18251 consensus_peers.cc:597] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.026937 16879 tablet_replica.cc:333] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:59.027689 16879 raft_consensus.cc:2243] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.028457 16879 raft_consensus.cc:2272] T 6b643cd863d14f36a3b9341312ef4770 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.031250 16879 tablet_replica.cc:333] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa: stopping tablet replica
I20260317 16:17:59.031822 16879 raft_consensus.cc:2243] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.032271 16879 raft_consensus.cc:2272] T 701d2829bc6a450a94470d7f852ecdc8 P 88b3a81306894b71bf2eacc88c770eaa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.062300 18251 consensus_peers.cc:597] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.070225 18781 consensus_peers.cc:597] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.101557 18781 consensus_peers.cc:597] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.152524 18781 consensus_peers.cc:597] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.169755 16879 tablet_server.cc:195] TabletServer@127.16.123.194:0 shutdown complete.
W20260317 16:17:59.190147 18781 consensus_peers.cc:597] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.222080 18251 consensus_peers.cc:597] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.234333 18251 consensus_peers.cc:597] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 -> Peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Couldn't send request to peer 88b3a81306894b71bf2eacc88c770eaa. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.278154 19880 raft_consensus.cc:493] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.279227 19880 raft_consensus.cc:515] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:59.283715 19880 leader_election.cc:290] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:59.288609 18249 leader_election.cc:336] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.290891 19880 raft_consensus.cc:493] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.291461 19880 raft_consensus.cc:515] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:59.293620 19880 leader_election.cc:290] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.298986 19892 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.299784 19892 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.302879 19892 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:59.303995 18251 leader_election.cc:336] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.304966 18251 leader_election.cc:304] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.304466 18282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
I20260317 16:17:59.308367 19892 raft_consensus.cc:493] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.309026 19880 raft_consensus.cc:2749] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:59.308991 19892 raft_consensus.cc:515] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
W20260317 16:17:59.309937 18251 leader_election.cc:336] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
W20260317 16:17:59.310268 18779 leader_election.cc:336] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.310819 18779 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.311446 19892 leader_election.cc:290] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.311969 19892 raft_consensus.cc:2749] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:17:59.317281 18249 leader_election.cc:336] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
W20260317 16:17:59.318302 18781 leader_election.cc:336] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.317900 18249 leader_election.cc:304] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.319229 19892 raft_consensus.cc:493] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.319900 19880 raft_consensus.cc:2749] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:17:59.319847 19892 raft_consensus.cc:515] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
W20260317 16:17:59.324301 18779 leader_election.cc:336] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.324735 18779 leader_election.cc:304] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.325608 19877 raft_consensus.cc:2749] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:17:59.326751 19892 leader_election.cc:290] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.328912 19892 raft_consensus.cc:493] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.329223 19877 raft_consensus.cc:493] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.329582 19892 raft_consensus.cc:515] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.329913 19877 raft_consensus.cc:515] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
W20260317 16:17:59.332006 18781 leader_election.cc:336] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.332041 19892 leader_election.cc:290] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:59.332229 19877 leader_election.cc:290] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.333734 18286 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "021d78130690452ba094f4a392fb4378" is_pre_election: true
W20260317 16:17:59.338274 18781 leader_election.cc:336] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
W20260317 16:17:59.352370 18779 leader_election.cc:336] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.352835 18779 leader_election.cc:304] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.354888 19877 raft_consensus.cc:2749] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:17:59.358739 18779 leader_election.cc:336] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.359609 18779 leader_election.cc:304] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
W20260317 16:17:59.360534 18779 leader_election.cc:336] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.360935 18779 leader_election.cc:304] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.361981 19892 raft_consensus.cc:2749] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:59.362927 19892 raft_consensus.cc:2749] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:59.379452 16879 tablet_server.cc:178] TabletServer@127.16.123.195:0 shutting down...
I20260317 16:17:59.383915 19880 raft_consensus.cc:493] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.384573 19880 raft_consensus.cc:515] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.388675 19880 leader_election.cc:290] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
W20260317 16:17:59.392439 18782 consensus_peers.cc:597] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.394873 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" is_pre_election: true
I20260317 16:17:59.396088 18812 raft_consensus.cc:2468] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 1.
I20260317 16:17:59.397744 18249 leader_election.cc:304] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:59.399295 19891 raft_consensus.cc:2804] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:17:59.400100 19891 raft_consensus.cc:493] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.400529 19891 raft_consensus.cc:3060] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:59.402432 19880 raft_consensus.cc:493] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.408246 19910 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.408994 19910 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.403249 19880 raft_consensus.cc:515] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:59.411583 19910 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:59.421482 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe3f3aabd9b444868c9732a8e5fa7440" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" is_pre_election: true
W20260317 16:17:59.429836 18083 auto_rebalancer.cc:798] Removing replace marker failed: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:59.430289 18083 auto_rebalancer.cc:806] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:59.430619 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:59.432653 18251 leader_election.cc:336] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.416986 19880 leader_election.cc:290] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:17:59.444078 18782 consensus_peers.cc:597] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.444784 18249 leader_election.cc:336] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
W20260317 16:17:59.454342 18083 auto_rebalancer.cc:798] Removing replace marker failed: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:59.454849 18083 auto_rebalancer.cc:806] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:59.455144 18083 auto_rebalancer.cc:285] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 16:17:59.471750 19891 raft_consensus.cc:515] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.474076 19917 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.474895 19891 leader_election.cc:290] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:59.493014 19917 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.507725 19917 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:59.514272 19919 raft_consensus.cc:493] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.517335 19919 raft_consensus.cc:515] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:59.529893 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eb8a46e02034bd199afdbe545379351" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
I20260317 16:17:59.530558 18812 raft_consensus.cc:3060] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:17:59.532008 18817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f2b31d108e461d932c2477e0f78fd0" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" is_pre_election: true
I20260317 16:17:59.532438 19919 leader_election.cc:290] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.533272 19919 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.533789 19919 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.535928 19919 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
W20260317 16:17:59.538125 18782 consensus_peers.cc:597] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.539628 18249 leader_election.cc:336] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.540495 18249 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.543730 19917 raft_consensus.cc:2749] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:17:59.546563 18782 consensus_peers.cc:597] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:17:59.547240 18249 leader_election.cc:336] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.547708 18249 leader_election.cc:304] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.548621 19917 raft_consensus.cc:2749] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:17:59.551951 18817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" is_pre_election: true
I20260317 16:17:59.552160 18812 raft_consensus.cc:2468] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
I20260317 16:17:59.552597 18817 raft_consensus.cc:2468] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 2.
W20260317 16:17:59.553718 18251 leader_election.cc:336] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.555382 18249 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:59.564505 19919 raft_consensus.cc:2804] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:17:59.564492 18249 leader_election.cc:304] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:59.564973 19919 raft_consensus.cc:493] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.565704 19919 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:59.565917 19910 raft_consensus.cc:2804] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:17:59.566419 19910 raft_consensus.cc:697] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:59.567310 19910 consensus_queue.cc:237] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
W20260317 16:17:59.567847 18249 leader_election.cc:336] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
W20260317 16:17:59.568951 18249 leader_election.cc:336] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.569382 18249 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 381cc8622e1f4a53a7ff2b23ed2bcd38, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.570067 19880 raft_consensus.cc:2749] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:17:59.570272 18249 leader_election.cc:336] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.570832 18249 leader_election.cc:304] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
W20260317 16:17:59.576057 18782 consensus_peers.cc:597] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.576342 19919 raft_consensus.cc:515] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.580498 19919 leader_election.cc:290] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: Requested vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 381cc8622e1f4a53a7ff2b23ed2bcd38 (127.16.123.196:40243)
I20260317 16:17:59.581907 19919 raft_consensus.cc:2749] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:17:59.582945 19914 raft_consensus.cc:493] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.583581 19914 raft_consensus.cc:515] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } attrs { replace: false } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.584713 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:17:59.586262 16879 tablet_replica.cc:333] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.585964 19914 leader_election.cc:290] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:59.587335 16879 raft_consensus.cc:2243] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.588043 16879 raft_consensus.cc:2272] T bfaeab6c75f94e128924d831ee8ef55b P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.589021 18782 leader_election.cc:336] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 16:17:59.591449 18779 leader_election.cc:336] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.591308 16879 tablet_replica.cc:333] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.592022 18779 leader_election.cc:304] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.592854 19914 raft_consensus.cc:2749] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:17:59.593066 16879 raft_consensus.cc:2243] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.594019 16879 pending_rounds.cc:70] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: Trying to abort 1 pending ops.
I20260317 16:17:59.594239 16879 pending_rounds.cc:77] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378: Aborting op as it isn't in flight: id { term: 2 index: 5 } timestamp: 7265338489131827200 op_type: NO_OP noop_request { }
I20260317 16:17:59.594547 16879 raft_consensus.cc:2889] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 16:17:59.594846 16879 raft_consensus.cc:2272] T 0eb8a46e02034bd199afdbe545379351 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.595830 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a6f1b891ac4c3492ae2548ff2fc357" candidate_uuid: "021d78130690452ba094f4a392fb4378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38"
W20260317 16:17:59.596305 18249 leader_election.cc:336] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.596446 18812 raft_consensus.cc:3060] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:17:59.600260 16879 tablet_replica.cc:333] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.601771 16879 raft_consensus.cc:2243] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.605515 18812 raft_consensus.cc:2468] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021d78130690452ba094f4a392fb4378 in term 3.
I20260317 16:17:59.608350 16879 raft_consensus.cc:2272] T a8ed2db954354e669de7909d315142aa P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.611794 19914 raft_consensus.cc:493] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.612370 19914 raft_consensus.cc:515] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.613222 16879 tablet_replica.cc:333] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.614028 18249 leader_election.cc:304] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021d78130690452ba094f4a392fb4378, 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters:
I20260317 16:17:59.615048 19910 raft_consensus.cc:2804] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:17:59.616072 19910 raft_consensus.cc:697] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Becoming Leader. State: Replica: 021d78130690452ba094f4a392fb4378, State: Running, Role: LEADER
I20260317 16:17:59.618137 19910 consensus_queue.cc:237] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
W20260317 16:17:59.621197 18782 consensus_peers.cc:597] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.623723 19914 leader_election.cc:290] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.615015 16879 raft_consensus.cc:2243] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.627817 16879 raft_consensus.cc:2272] T c5389f9d0bc44e87a3f5897fb4b550ef P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.628624 18779 leader_election.cc:336] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
W20260317 16:17:59.630159 18249 leader_election.cc:336] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.631274 16879 tablet_replica.cc:333] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
W20260317 16:17:59.631744 18781 leader_election.cc:336] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.632174 18781 leader_election.cc:304] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.633939 19915 raft_consensus.cc:2749] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:17:59.634127 16879 raft_consensus.cc:2243] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.634845 16879 raft_consensus.cc:2272] T f1e740c77f5f4cb6916938c76fd04455 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.640173 16879 tablet_replica.cc:333] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.642593 16879 raft_consensus.cc:2243] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.643689 16879 raft_consensus.cc:2272] T 8d3017628c5244c3a8bb091d0a4768fd P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.649499 16879 tablet_replica.cc:333] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
W20260317 16:17:59.649758 18782 consensus_peers.cc:597] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.650310 16879 raft_consensus.cc:2243] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.650978 16879 raft_consensus.cc:2272] T 4ee9125e28d14f3c8f458dda86c4ee15 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.653563 16879 tablet_replica.cc:333] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.654634 16879 raft_consensus.cc:2243] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.655191 16879 raft_consensus.cc:2272] T 44eefa978130466ead1b5828a0e0c4f3 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.656680 19915 raft_consensus.cc:493] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.657253 19915 raft_consensus.cc:515] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.658260 16879 tablet_replica.cc:333] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.659209 16879 raft_consensus.cc:2243] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.659864 19915 leader_election.cc:290] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:59.659955 16879 raft_consensus.cc:2272] T e2a036705e4a4da5a5392d119edb244e P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.663405 16879 tablet_replica.cc:333] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
W20260317 16:17:59.663545 18779 leader_election.cc:336] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.664168 16879 raft_consensus.cc:2243] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
W20260317 16:17:59.664353 18782 leader_election.cc:336] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 16:17:59.664839 18782 leader_election.cc:304] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.665336 16879 raft_consensus.cc:2272] T 1897e8c1602b4aeda93d9b8207b3ac45 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.666113 19915 raft_consensus.cc:2749] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:59.668114 16879 tablet_replica.cc:333] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.668705 16879 raft_consensus.cc:2243] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.669209 16879 raft_consensus.cc:2272] T f162d501437442c1ad448749ff6b9d77 P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.671468 16879 tablet_replica.cc:333] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.672312 16879 raft_consensus.cc:2243] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.673295 16879 raft_consensus.cc:2272] T ab7a4647163b461f87a0a249491e566c P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.675333 16879 tablet_replica.cc:333] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.676918 16879 raft_consensus.cc:2243] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.677848 16879 raft_consensus.cc:2272] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.680054 16879 tablet_replica.cc:333] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.680994 16879 raft_consensus.cc:2243] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.681890 16879 raft_consensus.cc:2272] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.684062 16879 tablet_replica.cc:333] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.684806 16879 raft_consensus.cc:2243] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.685695 16879 raft_consensus.cc:2272] T 9545c351b28449c2a44da8b9a94c7f11 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.687752 16879 tablet_replica.cc:333] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.688324 16879 raft_consensus.cc:2243] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.688774 16879 raft_consensus.cc:2272] T 9e145759d6de4334931c0039a97abdd8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.690613 16879 tablet_replica.cc:333] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.691216 16879 raft_consensus.cc:2243] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.691639 16879 raft_consensus.cc:2272] T 24063e0e6d6a4826899168d2524d5abc P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.693658 16879 tablet_replica.cc:333] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.694195 16879 raft_consensus.cc:2243] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.694839 16879 raft_consensus.cc:2272] T 94f2b31d108e461d932c2477e0f78fd0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.697423 16879 tablet_replica.cc:333] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.698127 16879 raft_consensus.cc:2243] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.698683 16879 raft_consensus.cc:2272] T b451ac456d424629ae5ce6040cb15bd2 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.700714 16879 tablet_replica.cc:333] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.701210 16879 raft_consensus.cc:2243] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.701876 16879 raft_consensus.cc:2272] T 17fe426def0d4f378a44ce5ec328f2e0 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.703707 16879 tablet_replica.cc:333] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.704391 16879 raft_consensus.cc:2243] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.704922 16879 raft_consensus.cc:2272] T 9246439ff23646f8abdf7bba7f3a9172 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.707764 16879 tablet_replica.cc:333] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.708369 16879 raft_consensus.cc:2243] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.709319 16879 raft_consensus.cc:2272] T 702896c0cf9948d68e4c4148743bc5c8 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.711351 16879 tablet_replica.cc:333] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.711907 16879 raft_consensus.cc:2243] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.712589 19915 raft_consensus.cc:493] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.712666 19910 raft_consensus.cc:493] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.712952 16879 raft_consensus.cc:2272] T 0b2a4d319ba9480b9c950e12b7329101 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.713250 19910 raft_consensus.cc:515] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } }
I20260317 16:17:59.713253 19915 raft_consensus.cc:515] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.715487 16879 tablet_replica.cc:333] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.715638 19915 leader_election.cc:290] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:59.716389 16879 raft_consensus.cc:2243] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.716547 19910 leader_election.cc:290] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:17:59.717484 16879 pending_rounds.cc:70] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: Trying to abort 1 pending ops.
I20260317 16:17:59.717718 16879 pending_rounds.cc:77] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378: Aborting op as it isn't in flight: id { term: 3 index: 6 } timestamp: 7265338489330442240 op_type: NO_OP noop_request { }
I20260317 16:17:59.719952 16879 raft_consensus.cc:2889] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20260317 16:17:59.720057 18782 leader_election.cc:336] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 16:17:59.720507 16879 raft_consensus.cc:2272] T 07a6f1b891ac4c3492ae2548ff2fc357 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.724088 18249 leader_election.cc:336] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
W20260317 16:17:59.724344 18779 leader_election.cc:336] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.724588 16879 tablet_replica.cc:333] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.724735 18779 leader_election.cc:304] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
W20260317 16:17:59.724953 18251 leader_election.cc:336] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.725418 18251 leader_election.cc:304] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 021d78130690452ba094f4a392fb4378; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:17:59.725778 16879 raft_consensus.cc:2243] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.725615 19915 raft_consensus.cc:2749] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:17:59.726207 19910 raft_consensus.cc:2749] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:59.726490 16879 raft_consensus.cc:2272] T 898420a8ccf84bbdba91710ca9c08ef1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.728497 16879 tablet_replica.cc:333] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.729094 16879 raft_consensus.cc:2243] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.729585 16879 raft_consensus.cc:2272] T 9844da16b03545c5a9e5534b29b4343c P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.731441 16879 tablet_replica.cc:333] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.731956 16879 raft_consensus.cc:2243] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.732362 16879 raft_consensus.cc:2272] T 5ded1f55ede142cbbde7c675a0f72d15 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.734110 16879 tablet_replica.cc:333] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.734655 16879 raft_consensus.cc:2243] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.735152 16879 raft_consensus.cc:2272] T fdf3aa784b554e5386dcc11e2550e2f5 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.736878 16879 tablet_replica.cc:333] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.737411 16879 raft_consensus.cc:2243] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.737821 16879 raft_consensus.cc:2272] T 519bc447dbc1455c844945ba63c96914 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.739655 16879 tablet_replica.cc:333] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.740237 16879 raft_consensus.cc:2243] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.741253 16879 raft_consensus.cc:2272] T 7aafcd49c9d54f548b25a51935f79523 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.744102 16879 tablet_replica.cc:333] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.744905 16879 raft_consensus.cc:2243] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.745653 16879 raft_consensus.cc:2272] T c94382dceadb48228dd3467bf673a11a P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.747778 16879 tablet_replica.cc:333] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.748328 16879 raft_consensus.cc:2243] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.749045 16879 raft_consensus.cc:2272] T ac85b776838146d9904478a4ef0ae824 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.749477 18782 consensus_peers.cc:597] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Couldn't send request to peer 021d78130690452ba094f4a392fb4378. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:17:59.751205 16879 tablet_replica.cc:333] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.751837 16879 raft_consensus.cc:2243] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.752288 16879 raft_consensus.cc:2272] T 07c5d59b5b404eec925d5c4681953edb P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.754153 16879 tablet_replica.cc:333] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.754727 16879 raft_consensus.cc:2243] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.755228 16879 raft_consensus.cc:2272] T dbff882fd6aa4aa0a32d72d1674abc76 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.756939 16879 tablet_replica.cc:333] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.757586 16879 raft_consensus.cc:2243] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.758332 16879 raft_consensus.cc:2272] T 84ded5b9676242a18139f88dc63af2b4 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.760170 16879 tablet_replica.cc:333] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.760722 16879 raft_consensus.cc:2243] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.761794 16879 raft_consensus.cc:2272] T 85f478d86a0949f381f0d88a657ba4e5 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.764340 16879 tablet_replica.cc:333] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.764905 16879 raft_consensus.cc:2243] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 4 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.765836 16879 raft_consensus.cc:2272] T 4a773cd94be84692a95e55317c98a95c P 021d78130690452ba094f4a392fb4378 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.765779 19915 raft_consensus.cc:493] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 94edf207909c4f8990bfe29ed1a87a6d)
I20260317 16:17:59.766528 19915 raft_consensus.cc:515] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } attrs { replace: true } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.768471 16879 tablet_replica.cc:333] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.769286 16879 raft_consensus.cc:2243] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.769883 16879 raft_consensus.cc:2272] T e6aca2493a994ed888cd663ddcf97582 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.770164 19915 leader_election.cc:290] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
I20260317 16:17:59.772513 16879 tablet_replica.cc:333] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.773306 16879 raft_consensus.cc:2243] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 LEADER]: Raft consensus shutting down.
W20260317 16:17:59.774302 18782 leader_election.cc:336] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 16:17:59.774536 16879 raft_consensus.cc:2272] T c85ea9795a2c43a38f1a9c3f3741db36 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260317 16:17:59.775712 18779 leader_election.cc:336] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:17:59.776193 18779 leader_election.cc:304] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 94edf207909c4f8990bfe29ed1a87a6d
W20260317 16:17:59.776964 18781 leader_election.cc:336] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
I20260317 16:17:59.777050 19915 raft_consensus.cc:2749] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:17:59.777412 16879 tablet_replica.cc:333] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.778196 16879 raft_consensus.cc:2243] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.778968 16879 raft_consensus.cc:2272] T fb31cad8b88240acb9c101027ea444c1 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.780725 16879 tablet_replica.cc:333] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.781365 16879 raft_consensus.cc:2243] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.782035 16879 raft_consensus.cc:2272] T 1cf03ff443ed41d3bb50d992c1f34372 P 021d78130690452ba094f4a392fb4378 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.784101 16879 tablet_replica.cc:333] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.784647 16879 raft_consensus.cc:2243] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.785334 16879 raft_consensus.cc:2272] T 55ecf327ed214b02bef87b8c05d54c26 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.787261 16879 tablet_replica.cc:333] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.787762 16879 raft_consensus.cc:2243] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.788203 16879 raft_consensus.cc:2272] T fe3f3aabd9b444868c9732a8e5fa7440 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.789947 16879 tablet_replica.cc:333] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.790407 16879 raft_consensus.cc:2243] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.791225 16879 raft_consensus.cc:2272] T 2f120096f51b4b3a992048069e2ae182 P 021d78130690452ba094f4a392fb4378 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.793358 16879 tablet_replica.cc:333] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.793955 16879 raft_consensus.cc:2243] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:17:59.794874 16879 raft_consensus.cc:2272] T 9f14079a4ea34cbda92828ca274134f4 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.797983 16879 tablet_replica.cc:333] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378: stopping tablet replica
I20260317 16:17:59.798460 16879 raft_consensus.cc:2243] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:17:59.798949 16879 raft_consensus.cc:2272] T 701d2829bc6a450a94470d7f852ecdc8 P 021d78130690452ba094f4a392fb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:17:59.889618 16879 tablet_server.cc:195] TabletServer@127.16.123.195:0 shutdown complete.
I20260317 16:17:59.917414 19892 raft_consensus.cc:493] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88b3a81306894b71bf2eacc88c770eaa)
I20260317 16:17:59.918058 19892 raft_consensus.cc:515] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "021d78130690452ba094f4a392fb4378" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 44851 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:17:59.925002 19892 leader_election.cc:290] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635), 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851)
W20260317 16:17:59.928350 18781 leader_election.cc:336] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
W20260317 16:17:59.930742 18782 leader_election.cc:336] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 021d78130690452ba094f4a392fb4378 (127.16.123.195:44851): Network error: Client connection negotiation failed: client connection to 127.16.123.195:44851: connect: Connection refused (error 111)
I20260317 16:17:59.931320 18782 leader_election.cc:304] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 021d78130690452ba094f4a392fb4378, 88b3a81306894b71bf2eacc88c770eaa
I20260317 16:17:59.932137 19892 raft_consensus.cc:2749] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:17:59.993458 16879 tablet_server.cc:178] TabletServer@127.16.123.196:0 shutting down...
I20260317 16:18:00.019353 19892 raft_consensus.cc:493] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88b3a81306894b71bf2eacc88c770eaa)
I20260317 16:18:00.020030 19892 raft_consensus.cc:515] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:18:00.027014 19892 leader_election.cc:290] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
W20260317 16:18:00.037990 18781 leader_election.cc:336] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
W20260317 16:18:00.048218 18779 leader_election.cc:336] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:18:00.048638 18779 leader_election.cc:304] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:18:00.049256 19892 raft_consensus.cc:2749] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:00.057618 16879 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 16:18:00.059242 16879 tablet_replica.cc:333] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.060254 16879 raft_consensus.cc:2243] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.061004 16879 raft_consensus.cc:2272] T 77f02a05d0ee4e73bafe3055cfcf02f2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.063915 16879 tablet_replica.cc:333] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.065086 16879 raft_consensus.cc:2243] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.065730 16879 raft_consensus.cc:2272] T 8c9d82ef74c442299f4a3cf04a5bb2d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.067777 16879 tablet_replica.cc:333] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.068396 16879 raft_consensus.cc:2243] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.069010 16879 raft_consensus.cc:2272] T 7aafcd49c9d54f548b25a51935f79523 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.070735 16879 tablet_replica.cc:333] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.071427 16879 raft_consensus.cc:2243] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.071970 16879 raft_consensus.cc:2272] T 1cf03ff443ed41d3bb50d992c1f34372 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.073691 16879 tablet_replica.cc:333] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.074250 16879 raft_consensus.cc:2243] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.074673 16879 raft_consensus.cc:2272] T 96d59cc0e53747428c5ff7f5178f11ca P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.077044 16879 tablet_replica.cc:333] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.077840 16879 raft_consensus.cc:2243] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.078538 16879 raft_consensus.cc:2272] T 1897e8c1602b4aeda93d9b8207b3ac45 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.081276 16879 tablet_replica.cc:333] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.082289 16879 raft_consensus.cc:2243] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.082896 16879 raft_consensus.cc:2272] T 75ecbd9bfe9e4fe4b1f5146d5029d1c0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.084748 16879 tablet_replica.cc:333] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.085534 16879 raft_consensus.cc:2243] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.086375 16879 raft_consensus.cc:2272] T 94f2b31d108e461d932c2477e0f78fd0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.089188 16879 tablet_replica.cc:333] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.090128 16879 raft_consensus.cc:2243] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.090768 16879 raft_consensus.cc:2272] T e390c786df0b48878244a08e889950a3 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:18:00.091898 18779 consensus_peers.cc:597] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 -> Peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Couldn't send request to peer 94edf207909c4f8990bfe29ed1a87a6d. Status: Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260317 16:18:00.093019 16879 tablet_replica.cc:333] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.093598 16879 raft_consensus.cc:2243] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.094267 16879 raft_consensus.cc:2272] T ab7a4647163b461f87a0a249491e566c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.098944 16879 tablet_replica.cc:333] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.100029 16879 raft_consensus.cc:2243] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.101174 16879 raft_consensus.cc:2272] T a8ed2db954354e669de7909d315142aa P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.103189 16879 tablet_replica.cc:333] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.104099 16879 raft_consensus.cc:2243] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.105118 16879 raft_consensus.cc:2272] T 702896c0cf9948d68e4c4148743bc5c8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.106967 16879 tablet_replica.cc:333] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.107532 16879 raft_consensus.cc:2243] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.108079 16879 raft_consensus.cc:2272] T 07a6f1b891ac4c3492ae2548ff2fc357 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.110167 16879 tablet_replica.cc:333] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.110875 16879 raft_consensus.cc:2243] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.111265 19892 raft_consensus.cc:493] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88b3a81306894b71bf2eacc88c770eaa)
I20260317 16:18:00.111447 16879 raft_consensus.cc:2272] T 0b2a4d319ba9480b9c950e12b7329101 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.111830 19892 raft_consensus.cc:515] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94edf207909c4f8990bfe29ed1a87a6d" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 45673 } } peers { permanent_uuid: "88b3a81306894b71bf2eacc88c770eaa" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35635 } } peers { permanent_uuid: "381cc8622e1f4a53a7ff2b23ed2bcd38" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 40243 } attrs { promote: false } }
I20260317 16:18:00.113746 19892 leader_election.cc:290] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673), 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635)
I20260317 16:18:00.114447 16879 tablet_replica.cc:333] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.115243 16879 raft_consensus.cc:2243] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.118274 16879 raft_consensus.cc:2272] T fdb770e745e0447faf1334e5d343ace0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260317 16:18:00.120504 18781 leader_election.cc:336] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88b3a81306894b71bf2eacc88c770eaa (127.16.123.194:35635): Network error: Client connection negotiation failed: client connection to 127.16.123.194:35635: connect: Connection refused (error 111)
W20260317 16:18:00.122221 18779 leader_election.cc:336] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94edf207909c4f8990bfe29ed1a87a6d (127.16.123.193:45673): Network error: Client connection negotiation failed: client connection to 127.16.123.193:45673: connect: Connection refused (error 111)
I20260317 16:18:00.122745 16879 tablet_replica.cc:333] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.123422 18779 leader_election.cc:304] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 381cc8622e1f4a53a7ff2b23ed2bcd38; no voters: 88b3a81306894b71bf2eacc88c770eaa, 94edf207909c4f8990bfe29ed1a87a6d
I20260317 16:18:00.124300 16879 raft_consensus.cc:2243] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.124488 19892 raft_consensus.cc:2749] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:00.125447 16879 raft_consensus.cc:2272] T 0eb8a46e02034bd199afdbe545379351 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.127846 16879 tablet_replica.cc:333] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.128620 16879 raft_consensus.cc:2243] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.129453 16879 raft_consensus.cc:2272] T 24063e0e6d6a4826899168d2524d5abc P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.131232 16879 tablet_replica.cc:333] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.131794 16879 raft_consensus.cc:2243] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.132606 16879 raft_consensus.cc:2272] T ac85b776838146d9904478a4ef0ae824 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.134573 16879 tablet_replica.cc:333] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.135419 16879 raft_consensus.cc:2243] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.136157 16879 raft_consensus.cc:2272] T bfaeab6c75f94e128924d831ee8ef55b P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.138510 16879 tablet_replica.cc:333] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.139219 16879 raft_consensus.cc:2243] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.139701 16879 raft_consensus.cc:2272] T 290a7a9eceeb4ea8a8ddb67d3e330082 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.142076 16879 tablet_replica.cc:333] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.144516 16879 raft_consensus.cc:2243] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.146678 16879 raft_consensus.cc:2272] T 9844da16b03545c5a9e5534b29b4343c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.149331 16879 tablet_replica.cc:333] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.150002 16879 raft_consensus.cc:2243] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.150498 16879 raft_consensus.cc:2272] T 4ee9125e28d14f3c8f458dda86c4ee15 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.152384 16879 tablet_replica.cc:333] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.152968 16879 raft_consensus.cc:2243] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.153367 16879 raft_consensus.cc:2272] T 519bc447dbc1455c844945ba63c96914 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.155180 16879 tablet_replica.cc:333] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.155740 16879 raft_consensus.cc:2243] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.156423 16879 raft_consensus.cc:2272] T 07c5d59b5b404eec925d5c4681953edb P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.158531 16879 tablet_replica.cc:333] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.160580 16879 raft_consensus.cc:2243] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.161401 16879 raft_consensus.cc:2272] T 09f47966c253483393878fddf97ac206 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.164048 16879 tablet_replica.cc:333] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.164695 16879 raft_consensus.cc:2243] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.165325 16879 raft_consensus.cc:2272] T 6b643cd863d14f36a3b9341312ef4770 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.167300 16879 tablet_replica.cc:333] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.167979 16879 raft_consensus.cc:2243] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.168452 16879 raft_consensus.cc:2272] T 87a9551ad4c24868805c90f4fa899ff0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.170318 16879 tablet_replica.cc:333] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.170976 16879 raft_consensus.cc:2243] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.171432 16879 raft_consensus.cc:2272] T 17fe426def0d4f378a44ce5ec328f2e0 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.173285 16879 tablet_replica.cc:333] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.173907 16879 raft_consensus.cc:2243] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.174340 16879 raft_consensus.cc:2272] T b451ac456d424629ae5ce6040cb15bd2 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.176141 16879 tablet_replica.cc:333] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.176694 16879 raft_consensus.cc:2243] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.177191 16879 raft_consensus.cc:2272] T 85f478d86a0949f381f0d88a657ba4e5 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.179195 16879 tablet_replica.cc:333] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.179759 16879 raft_consensus.cc:2243] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.180197 16879 raft_consensus.cc:2272] T c85ea9795a2c43a38f1a9c3f3741db36 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.182014 16879 tablet_replica.cc:333] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.182559 16879 raft_consensus.cc:2243] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.183077 16879 raft_consensus.cc:2272] T c94382dceadb48228dd3467bf673a11a P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.184893 16879 tablet_replica.cc:333] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.185446 16879 raft_consensus.cc:2243] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.185968 16879 raft_consensus.cc:2272] T fb31cad8b88240acb9c101027ea444c1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.188452 16879 tablet_replica.cc:333] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.189018 16879 raft_consensus.cc:2243] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.189478 16879 raft_consensus.cc:2272] T 84ded5b9676242a18139f88dc63af2b4 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.191437 16879 tablet_replica.cc:333] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.192039 16879 raft_consensus.cc:2243] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.192776 16879 raft_consensus.cc:2272] T d88548519a0f473680897b8a7fc1e9b6 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.194610 16879 tablet_replica.cc:333] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.195303 16879 raft_consensus.cc:2243] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.196071 16879 raft_consensus.cc:2272] T 47801c6cdf6f461181a89246716390d7 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.197912 16879 tablet_replica.cc:333] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.198463 16879 raft_consensus.cc:2243] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.199220 16879 raft_consensus.cc:2272] T 70f2fc6f0fd640fcb531a01d7881a614 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.200915 16879 tablet_replica.cc:333] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.201418 16879 raft_consensus.cc:2243] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.202164 16879 raft_consensus.cc:2272] T e6aca2493a994ed888cd663ddcf97582 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.203950 16879 tablet_replica.cc:333] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.204456 16879 raft_consensus.cc:2243] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.204921 16879 raft_consensus.cc:2272] T f77a63cdb5604ab493b7f5f9dc921763 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.206696 16879 tablet_replica.cc:333] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.207207 16879 raft_consensus.cc:2243] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.207938 16879 raft_consensus.cc:2272] T 55ecf327ed214b02bef87b8c05d54c26 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.209666 16879 tablet_replica.cc:333] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.210167 16879 raft_consensus.cc:2243] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 16:18:00.210556 16879 raft_consensus.cc:2272] T fe3f3aabd9b444868c9732a8e5fa7440 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.212338 16879 tablet_replica.cc:333] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.212819 16879 raft_consensus.cc:2243] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.213515 16879 raft_consensus.cc:2272] T 9e145759d6de4334931c0039a97abdd8 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.215231 16879 tablet_replica.cc:333] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.215682 16879 raft_consensus.cc:2243] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.216411 16879 raft_consensus.cc:2272] T 8284701fdaf640daa689a2534914514c P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.218083 16879 tablet_replica.cc:333] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38: stopping tablet replica
I20260317 16:18:00.218518 16879 raft_consensus.cc:2243] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.219504 16879 raft_consensus.cc:2272] T 898420a8ccf84bbdba91710ca9c08ef1 P 381cc8622e1f4a53a7ff2b23ed2bcd38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.289693 16879 tablet_server.cc:195] TabletServer@127.16.123.196:0 shutdown complete.
I20260317 16:18:00.361536 16879 master.cc:561] Master@127.16.123.254:39965 shutting down...
I20260317 16:18:00.410925 16879 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 1 LEADER]: Raft consensus shutting down.
I20260317 16:18:00.411623 16879 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 16:18:00.411970 16879 tablet_replica.cc:333] T 00000000000000000000000000000000 P 387423d8ffe1441b90cc64abbbf247c7: stopping tablet replica
I20260317 16:18:00.439651 16879 master.cc:583] Master@127.16.123.254:39965 shutdown complete.
[ OK ] LeaderRebalancerTest.AddTserver (54222 ms)
[ RUN ] LeaderRebalancerTest.RestartTserver
I20260317 16:18:00.559619 16879 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.123.254:35301
I20260317 16:18:00.561975 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:18:00.574237 19924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:00.574503 19923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:00.576077 16879 server_base.cc:1060] running on GCE node
W20260317 16:18:00.583465 19927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:00.585063 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:18:00.585407 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:18:00.585626 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764280585598 us; error 0 us; skew 500 ppm
I20260317 16:18:00.586452 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:18:00.597466 16879 webserver.cc:492] Webserver started at http://127.16.123.254:33579/ using document root <none> and password file <none>
I20260317 16:18:00.598523 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:18:00.598862 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:18:00.599277 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:18:00.601585 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/master-0-root/instance:
uuid: "aebfd8ce7b084fe1af5511a23409311f"
format_stamp: "Formatted at 2026-03-17 16:18:00 on dist-test-slave-kqc6"
I20260317 16:18:00.612702 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.003s
I20260317 16:18:00.618199 19931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:00.619876 16879 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260317 16:18:00.620253 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/master-0-root
uuid: "aebfd8ce7b084fe1af5511a23409311f"
format_stamp: "Formatted at 2026-03-17 16:18:00 on dist-test-slave-kqc6"
I20260317 16:18:00.620986 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:18:00.660209 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:18:00.661881 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:18:00.727965 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.254:35301
I20260317 16:18:00.728116 19982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.254:35301 every 8 connection(s)
I20260317 16:18:00.732195 19983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:00.743427 19983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Bootstrap starting.
I20260317 16:18:00.748067 19983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:00.752442 19983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: No bootstrap required, opened a new log
I20260317 16:18:00.756585 19983 raft_consensus.cc:359] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfd8ce7b084fe1af5511a23409311f" member_type: VOTER }
I20260317 16:18:00.757642 19983 raft_consensus.cc:385] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:00.758247 19983 raft_consensus.cc:740] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebfd8ce7b084fe1af5511a23409311f, State: Initialized, Role: FOLLOWER
I20260317 16:18:00.760425 19983 consensus_queue.cc:260] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfd8ce7b084fe1af5511a23409311f" member_type: VOTER }
I20260317 16:18:00.761169 19983 raft_consensus.cc:399] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 16:18:00.761432 19983 raft_consensus.cc:493] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 16:18:00.761734 19983 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:00.772060 19983 raft_consensus.cc:515] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfd8ce7b084fe1af5511a23409311f" member_type: VOTER }
I20260317 16:18:00.773972 19983 leader_election.cc:304] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aebfd8ce7b084fe1af5511a23409311f; no voters:
I20260317 16:18:00.777225 19983 leader_election.cc:290] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 16:18:00.777993 19986 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:00.779470 19986 raft_consensus.cc:697] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [term 1 LEADER]: Becoming Leader. State: Replica: aebfd8ce7b084fe1af5511a23409311f, State: Running, Role: LEADER
I20260317 16:18:00.780125 19986 consensus_queue.cc:237] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfd8ce7b084fe1af5511a23409311f" member_type: VOTER }
I20260317 16:18:00.780516 19983 sys_catalog.cc:565] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [sys.catalog]: configured and running, proceeding with master startup.
I20260317 16:18:00.793449 19987 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aebfd8ce7b084fe1af5511a23409311f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfd8ce7b084fe1af5511a23409311f" member_type: VOTER } }
I20260317 16:18:00.793798 19988 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aebfd8ce7b084fe1af5511a23409311f. Latest consensus state: current_term: 1 leader_uuid: "aebfd8ce7b084fe1af5511a23409311f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfd8ce7b084fe1af5511a23409311f" member_type: VOTER } }
I20260317 16:18:00.796947 19988 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [sys.catalog]: This master's current role is: LEADER
I20260317 16:18:00.797470 19987 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f [sys.catalog]: This master's current role is: LEADER
I20260317 16:18:00.801517 19992 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 16:18:00.821152 19992 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 16:18:00.829412 16879 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 16:18:00.841471 19992 catalog_manager.cc:1357] Generated new cluster ID: 89fd073c29844025bbb79b1ef03b2d45
I20260317 16:18:00.841815 19992 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 16:18:00.880180 19992 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 16:18:00.882676 19992 catalog_manager.cc:1514] Loading token signing keys...
I20260317 16:18:00.905164 19992 catalog_manager.cc:6044] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Generated new TSK 0
I20260317 16:18:00.906023 19992 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 16:18:00.963950 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:18:00.971197 20004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:00.972999 20005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:00.975616 20007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:00.976720 16879 server_base.cc:1060] running on GCE node
I20260317 16:18:00.978227 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:18:00.978495 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:18:00.978966 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764280978931 us; error 0 us; skew 500 ppm
I20260317 16:18:00.979735 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:18:00.982621 16879 webserver.cc:492] Webserver started at http://127.16.123.193:46437/ using document root <none> and password file <none>
I20260317 16:18:00.983213 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:18:00.983430 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:18:00.983737 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:18:00.986112 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/instance:
uuid: "9a3afb098d6a45f0b405566771a1157b"
format_stamp: "Formatted at 2026-03-17 16:18:00 on dist-test-slave-kqc6"
I20260317 16:18:00.992479 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260317 16:18:00.996785 20012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:00.998023 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260317 16:18:00.998420 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-0-root
uuid: "9a3afb098d6a45f0b405566771a1157b"
format_stamp: "Formatted at 2026-03-17 16:18:00 on dist-test-slave-kqc6"
I20260317 16:18:00.998736 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:18:01.023214 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:18:01.024585 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:18:01.026290 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:18:01.028753 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:18:01.029013 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.029277 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:18:01.029489 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.096241 20074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.193:40319 every 8 connection(s)
I20260317 16:18:01.096881 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.193:40319
I20260317 16:18:01.103169 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:18:01.113703 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:01.114399 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:01.118400 16879 server_base.cc:1060] running on GCE node
I20260317 16:18:01.121845 20075 heartbeater.cc:344] Connected to a master server at 127.16.123.254:35301
I20260317 16:18:01.122329 20075 heartbeater.cc:461] Registering TS with master...
W20260317 16:18:01.122272 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:01.123620 20075 heartbeater.cc:507] Master 127.16.123.254:35301 requested a full tablet report, sending...
I20260317 16:18:01.124876 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:18:01.125430 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:18:01.125818 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764281125789 us; error 0 us; skew 500 ppm
I20260317 16:18:01.126706 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:18:01.126678 19948 ts_manager.cc:194] Registered new tserver with Master: 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:01.129474 19948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46676
I20260317 16:18:01.132469 16879 webserver.cc:492] Webserver started at http://127.16.123.194:36817/ using document root <none> and password file <none>
I20260317 16:18:01.133116 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:18:01.133348 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:18:01.133673 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:18:01.135221 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/instance:
uuid: "795c49e9595f45bdb8d6b568498abf1f"
format_stamp: "Formatted at 2026-03-17 16:18:01 on dist-test-slave-kqc6"
I20260317 16:18:01.143723 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260317 16:18:01.149685 20087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.150738 16879 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260317 16:18:01.151118 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-1-root
uuid: "795c49e9595f45bdb8d6b568498abf1f"
format_stamp: "Formatted at 2026-03-17 16:18:01 on dist-test-slave-kqc6"
I20260317 16:18:01.151391 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:18:01.169915 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:18:01.171633 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:18:01.173676 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:18:01.176513 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:18:01.176959 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.177560 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:18:01.177922 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.241683 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.194:35679
I20260317 16:18:01.241845 20149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.194:35679 every 8 connection(s)
I20260317 16:18:01.247900 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:18:01.261457 20153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:01.262512 20154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:01.268822 20150 heartbeater.cc:344] Connected to a master server at 127.16.123.254:35301
I20260317 16:18:01.269290 20150 heartbeater.cc:461] Registering TS with master...
I20260317 16:18:01.271663 20150 heartbeater.cc:507] Master 127.16.123.254:35301 requested a full tablet report, sending...
I20260317 16:18:01.272853 16879 server_base.cc:1060] running on GCE node
W20260317 16:18:01.273808 20156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:01.275413 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 16:18:01.275918 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:18:01.276154 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764281276108 us; error 0 us; skew 500 ppm
I20260317 16:18:01.276247 19948 ts_manager.cc:194] Registered new tserver with Master: 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:01.277253 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:18:01.278890 19948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46692
I20260317 16:18:01.281701 16879 webserver.cc:492] Webserver started at http://127.16.123.195:39555/ using document root <none> and password file <none>
I20260317 16:18:01.282491 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:18:01.282733 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:18:01.283043 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:18:01.284140 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/instance:
uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
format_stamp: "Formatted at 2026-03-17 16:18:01 on dist-test-slave-kqc6"
I20260317 16:18:01.289536 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260317 16:18:01.293855 20161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.294864 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260317 16:18:01.295203 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-2-root
uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
format_stamp: "Formatted at 2026-03-17 16:18:01 on dist-test-slave-kqc6"
I20260317 16:18:01.295508 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:18:01.317255 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:18:01.318578 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:18:01.320848 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:18:01.323451 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:18:01.323710 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.323975 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:18:01.324148 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.385977 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.195:36885
I20260317 16:18:01.386085 20223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.195:36885 every 8 connection(s)
I20260317 16:18:01.391369 16879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 16:18:01.401240 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:01.401696 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 16:18:01.407491 20230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 16:18:01.407694 16879 server_base.cc:1060] running on GCE node
I20260317 16:18:01.409552 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260317 16:18:01.409441 20224 heartbeater.cc:344] Connected to a master server at 127.16.123.254:35301
W20260317 16:18:01.410036 16879 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 16:18:01.410418 16879 hybrid_clock.cc:648] HybridClock initialized: now 1773764281410398 us; error 0 us; skew 500 ppm
I20260317 16:18:01.410437 20224 heartbeater.cc:461] Registering TS with master...
I20260317 16:18:01.411823 20224 heartbeater.cc:507] Master 127.16.123.254:35301 requested a full tablet report, sending...
I20260317 16:18:01.411881 16879 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 16:18:01.415695 19948 ts_manager.cc:194] Registered new tserver with Master: 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:01.416848 16879 webserver.cc:492] Webserver started at http://127.16.123.196:44889/ using document root <none> and password file <none>
I20260317 16:18:01.417762 16879 fs_manager.cc:362] Metadata directory not provided
I20260317 16:18:01.418033 16879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 16:18:01.418201 19948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46702
I20260317 16:18:01.418383 16879 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 16:18:01.419886 16879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-3-root/instance:
uuid: "d8ed4b12dc654776aa7e64c0fa224321"
format_stamp: "Formatted at 2026-03-17 16:18:01 on dist-test-slave-kqc6"
I20260317 16:18:01.425774 16879 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260317 16:18:01.431198 20235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.432195 16879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260317 16:18:01.432530 16879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-3-root
uuid: "d8ed4b12dc654776aa7e64c0fa224321"
format_stamp: "Formatted at 2026-03-17 16:18:01 on dist-test-slave-kqc6"
I20260317 16:18:01.432812 16879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskwlTXSV/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1773764160731778-16879-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 16:18:01.458235 16879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 16:18:01.459707 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 16:18:01.461251 16879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 16:18:01.463800 16879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 16:18:01.464003 16879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 16:18:01.464598 16879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 16:18:01.465010 16879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260317 16:18:01.531234 16879 rpc_server.cc:307] RPC server started. Bound to: 127.16.123.196:43213
I20260317 16:18:01.531576 20297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.123.196:43213 every 8 connection(s)
I20260317 16:18:01.548177 20298 heartbeater.cc:344] Connected to a master server at 127.16.123.254:35301
I20260317 16:18:01.548712 20298 heartbeater.cc:461] Registering TS with master...
I20260317 16:18:01.549501 20298 heartbeater.cc:507] Master 127.16.123.254:35301 requested a full tablet report, sending...
I20260317 16:18:01.551673 19948 ts_manager.cc:194] Registered new tserver with Master: d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:01.552192 16879 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.015727005s
I20260317 16:18:01.553268 19948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46714
I20260317 16:18:01.587841 19948 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46722:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 16:18:01.814126 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:01.954505 20115 tablet_service.cc:1511] Processing CreateTablet for tablet 1234018d42ac449782301c404118e5bb (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260317 16:18:01.956677 20115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1234018d42ac449782301c404118e5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.957082 20114 tablet_service.cc:1511] Processing CreateTablet for tablet e98310d917124048971895e5cd5c71f1 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260317 16:18:01.958686 20112 tablet_service.cc:1511] Processing CreateTablet for tablet a07475a709344f5e812eea08aa4113cd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260317 16:18:01.963037 20114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98310d917124048971895e5cd5c71f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.960083 20111 tablet_service.cc:1511] Processing CreateTablet for tablet 49737061a2c940b9982aa52ccdf66e90 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260317 16:18:01.964432 20185 tablet_service.cc:1511] Processing CreateTablet for tablet 4c55cdb999654153a4cb90e34cb77217 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260317 16:18:01.964375 20186 tablet_service.cc:1511] Processing CreateTablet for tablet a07475a709344f5e812eea08aa4113cd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260317 16:18:01.960347 20113 tablet_service.cc:1511] Processing CreateTablet for tablet 5875427d66a54f92898d04d79d1bc6de (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260317 16:18:01.966547 20111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49737061a2c940b9982aa52ccdf66e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.961982 20187 tablet_service.cc:1511] Processing CreateTablet for tablet 5875427d66a54f92898d04d79d1bc6de (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260317 16:18:01.971995 20184 tablet_service.cc:1511] Processing CreateTablet for tablet 629f02fa987f4ce2a3b1522ee599490b (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260317 16:18:01.961982 20188 tablet_service.cc:1511] Processing CreateTablet for tablet 1234018d42ac449782301c404118e5bb (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260317 16:18:01.967036 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c55cdb999654153a4cb90e34cb77217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.972424 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5875427d66a54f92898d04d79d1bc6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.961961 20110 tablet_service.cc:1511] Processing CreateTablet for tablet 629f02fa987f4ce2a3b1522ee599490b (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260317 16:18:01.975015 20112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07475a709344f5e812eea08aa4113cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.975713 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5875427d66a54f92898d04d79d1bc6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.980445 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629f02fa987f4ce2a3b1522ee599490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.961601 20189 tablet_service.cc:1511] Processing CreateTablet for tablet 8720c3d1b3994abeba1e121c1cf99275 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260317 16:18:01.984064 20189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8720c3d1b3994abeba1e121c1cf99275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.985534 20186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07475a709344f5e812eea08aa4113cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.986619 20184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629f02fa987f4ce2a3b1522ee599490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.990208 20188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1234018d42ac449782301c404118e5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:01.993039 20109 tablet_service.cc:1511] Processing CreateTablet for tablet 6d13fb540668437094ca4190aa5dd7ec (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260317 16:18:01.999011 20183 tablet_service.cc:1511] Processing CreateTablet for tablet 6d13fb540668437094ca4190aa5dd7ec (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260317 16:18:02.000087 20109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d13fb540668437094ca4190aa5dd7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.001463 20183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d13fb540668437094ca4190aa5dd7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.021858 20040 tablet_service.cc:1511] Processing CreateTablet for tablet 8720c3d1b3994abeba1e121c1cf99275 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260317 16:18:02.024150 20108 tablet_service.cc:1511] Processing CreateTablet for tablet 352087b67c9f40f4a6fa365aa66b8f13 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260317 16:18:02.025929 20182 tablet_service.cc:1511] Processing CreateTablet for tablet a29e5bafc1d14cf4932e2a0489026dd2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260317 16:18:02.030452 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e5bafc1d14cf4932e2a0489026dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.036723 20263 tablet_service.cc:1511] Processing CreateTablet for tablet 8720c3d1b3994abeba1e121c1cf99275 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260317 16:18:02.040649 20262 tablet_service.cc:1511] Processing CreateTablet for tablet e98310d917124048971895e5cd5c71f1 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260317 16:18:02.044559 20261 tablet_service.cc:1511] Processing CreateTablet for tablet 5875427d66a54f92898d04d79d1bc6de (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260317 16:18:02.067512 20318 tablet_bootstrap.cc:492] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:02.067639 20258 tablet_service.cc:1511] Processing CreateTablet for tablet 629f02fa987f4ce2a3b1522ee599490b (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260317 16:18:02.069233 20260 tablet_service.cc:1511] Processing CreateTablet for tablet 49737061a2c940b9982aa52ccdf66e90 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260317 16:18:02.070272 20259 tablet_service.cc:1511] Processing CreateTablet for tablet 4c55cdb999654153a4cb90e34cb77217 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260317 16:18:02.064126 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352087b67c9f40f4a6fa365aa66b8f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.079119 20263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8720c3d1b3994abeba1e121c1cf99275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.079519 20107 tablet_service.cc:1511] Processing CreateTablet for tablet ef04e628857445e2a7a7cd12a684908e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260317 16:18:02.087927 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef04e628857445e2a7a7cd12a684908e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.088201 20181 tablet_service.cc:1511] Processing CreateTablet for tablet ef04e628857445e2a7a7cd12a684908e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260317 16:18:02.089916 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef04e628857445e2a7a7cd12a684908e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.079120 20040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8720c3d1b3994abeba1e121c1cf99275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.092896 20039 tablet_service.cc:1511] Processing CreateTablet for tablet 1234018d42ac449782301c404118e5bb (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260317 16:18:02.093066 20038 tablet_service.cc:1511] Processing CreateTablet for tablet e98310d917124048971895e5cd5c71f1 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260317 16:18:02.096614 20037 tablet_service.cc:1511] Processing CreateTablet for tablet a07475a709344f5e812eea08aa4113cd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260317 16:18:02.100895 20036 tablet_service.cc:1511] Processing CreateTablet for tablet 49737061a2c940b9982aa52ccdf66e90 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260317 16:18:02.104772 20257 tablet_service.cc:1511] Processing CreateTablet for tablet 352087b67c9f40f4a6fa365aa66b8f13 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260317 16:18:02.106185 20256 tablet_service.cc:1511] Processing CreateTablet for tablet a29e5bafc1d14cf4932e2a0489026dd2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260317 16:18:02.104025 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1234018d42ac449782301c404118e5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.108844 20035 tablet_service.cc:1511] Processing CreateTablet for tablet 4c55cdb999654153a4cb90e34cb77217 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260317 16:18:02.110183 20255 tablet_service.cc:1511] Processing CreateTablet for tablet ef04e628857445e2a7a7cd12a684908e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260317 16:18:02.112128 20032 tablet_service.cc:1511] Processing CreateTablet for tablet a29e5bafc1d14cf4932e2a0489026dd2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260317 16:18:02.113559 20033 tablet_service.cc:1511] Processing CreateTablet for tablet 352087b67c9f40f4a6fa365aa66b8f13 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260317 16:18:02.110450 20034 tablet_service.cc:1511] Processing CreateTablet for tablet 6d13fb540668437094ca4190aa5dd7ec (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260317 16:18:02.117545 20261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5875427d66a54f92898d04d79d1bc6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.121608 20258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629f02fa987f4ce2a3b1522ee599490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.122606 20257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352087b67c9f40f4a6fa365aa66b8f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.110456 20035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c55cdb999654153a4cb90e34cb77217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.127171 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98310d917124048971895e5cd5c71f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.127228 20259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c55cdb999654153a4cb90e34cb77217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.129956 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e5bafc1d14cf4932e2a0489026dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.135382 20075 heartbeater.cc:499] Master 127.16.123.254:35301 was elected leader, sending a full tablet report...
I20260317 16:18:02.136126 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef04e628857445e2a7a7cd12a684908e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.139616 20260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49737061a2c940b9982aa52ccdf66e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.143256 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07475a709344f5e812eea08aa4113cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.145726 20032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29e5bafc1d14cf4932e2a0489026dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.147558 20034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d13fb540668437094ca4190aa5dd7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.151151 20033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352087b67c9f40f4a6fa365aa66b8f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.154722 20036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49737061a2c940b9982aa52ccdf66e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.151432 20262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98310d917124048971895e5cd5c71f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.200526 20318 tablet_bootstrap.cc:654] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.220276 20320 tablet_bootstrap.cc:492] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:02.227208 20322 tablet_bootstrap.cc:492] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:02.248509 20322 tablet_bootstrap.cc:654] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.252365 20321 tablet_bootstrap.cc:492] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:02.282948 20150 heartbeater.cc:499] Master 127.16.123.254:35301 was elected leader, sending a full tablet report...
I20260317 16:18:02.299157 20320 tablet_bootstrap.cc:654] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.302110 20321 tablet_bootstrap.cc:654] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.331262 20320 tablet_bootstrap.cc:492] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:02.332074 20320 ts_tablet_manager.cc:1403] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.112s user 0.012s sys 0.005s
I20260317 16:18:02.332787 20321 tablet_bootstrap.cc:492] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:02.333321 20321 ts_tablet_manager.cc:1403] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.018s
I20260317 16:18:02.333982 20322 tablet_bootstrap.cc:492] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:02.334502 20322 ts_tablet_manager.cc:1403] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.108s user 0.040s sys 0.037s
I20260317 16:18:02.335345 20320 raft_consensus.cc:359] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.336362 20320 raft_consensus.cc:385] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:02.336717 20320 raft_consensus.cc:740] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:02.337543 20320 consensus_queue.cc:260] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.337775 20322 raft_consensus.cc:359] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:02.336001 20321 raft_consensus.cc:359] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.339114 20322 raft_consensus.cc:385] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:02.339928 20322 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:02.339283 20321 raft_consensus.cc:385] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:02.341063 20322 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:02.342224 20321 raft_consensus.cc:740] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:02.432941 20039 tablet_service.cc:1511] Processing CreateTablet for tablet a663a988cc754c24b227bb9bc85540db (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260317 16:18:02.439169 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a663a988cc754c24b227bb9bc85540db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.453716 20113 tablet_service.cc:1511] Processing CreateTablet for tablet a663a988cc754c24b227bb9bc85540db (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260317 16:18:02.456840 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a663a988cc754c24b227bb9bc85540db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.470233 20224 heartbeater.cc:499] Master 127.16.123.254:35301 was elected leader, sending a full tablet report...
I20260317 16:18:02.487885 20321 consensus_queue.cc:260] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.533170 20037 tablet_service.cc:1511] Processing CreateTablet for tablet 1ea78aa57aaf4f86b53cd07ab29faa49 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260317 16:18:02.535996 20320 ts_tablet_manager.cc:1434] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.202s user 0.010s sys 0.003s
I20260317 16:18:02.543460 20298 heartbeater.cc:499] Master 127.16.123.254:35301 was elected leader, sending a full tablet report...
I20260317 16:18:02.546739 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 1ea78aa57aaf4f86b53cd07ab29faa49 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260317 16:18:02.548846 20320 tablet_bootstrap.cc:492] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:02.549105 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea78aa57aaf4f86b53cd07ab29faa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.553560 20318 tablet_bootstrap.cc:492] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:02.554445 20318 ts_tablet_manager.cc:1403] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.488s user 0.018s sys 0.012s
I20260317 16:18:02.558447 20184 tablet_service.cc:1511] Processing CreateTablet for tablet 41e4ff5d2d614a199e8e8a314b43bb7c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260317 16:18:02.566394 20184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e4ff5d2d614a199e8e8a314b43bb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.568718 20259 tablet_service.cc:1511] Processing CreateTablet for tablet 41e4ff5d2d614a199e8e8a314b43bb7c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260317 16:18:02.570956 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea78aa57aaf4f86b53cd07ab29faa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.524953 20187 tablet_service.cc:1511] Processing CreateTablet for tablet a663a988cc754c24b227bb9bc85540db (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260317 16:18:02.573167 20259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e4ff5d2d614a199e8e8a314b43bb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.578334 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a663a988cc754c24b227bb9bc85540db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.558624 20107 tablet_service.cc:1511] Processing CreateTablet for tablet 1ea78aa57aaf4f86b53cd07ab29faa49 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260317 16:18:02.581529 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea78aa57aaf4f86b53cd07ab29faa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.584299 20318 raft_consensus.cc:359] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:02.585628 20318 raft_consensus.cc:385] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:02.586062 20318 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:02.586619 20321 ts_tablet_manager.cc:1434] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.253s user 0.036s sys 0.023s
I20260317 16:18:02.587548 20322 ts_tablet_manager.cc:1434] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.253s user 0.015s sys 0.008s
I20260317 16:18:02.588783 20322 tablet_bootstrap.cc:492] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:02.595270 20321 tablet_bootstrap.cc:492] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:02.593958 20318 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:02.620453 20035 tablet_service.cc:1511] Processing CreateTablet for tablet 41e4ff5d2d614a199e8e8a314b43bb7c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260317 16:18:02.622066 20035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e4ff5d2d614a199e8e8a314b43bb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.638228 20320 tablet_bootstrap.cc:654] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.644845 20318 ts_tablet_manager.cc:1434] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.085s user 0.009s sys 0.004s
I20260317 16:18:02.644519 20108 tablet_service.cc:1511] Processing CreateTablet for tablet 9e6529a0cb514111b1bfd58537d255d2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260317 16:18:02.646848 20318 tablet_bootstrap.cc:492] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:02.650153 20187 tablet_service.cc:1511] Processing CreateTablet for tablet 9e6529a0cb514111b1bfd58537d255d2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260317 16:18:02.650736 20257 tablet_service.cc:1511] Processing CreateTablet for tablet 9e6529a0cb514111b1bfd58537d255d2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260317 16:18:02.659885 20257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6529a0cb514111b1bfd58537d255d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.653308 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6529a0cb514111b1bfd58537d255d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.679896 20322 tablet_bootstrap.cc:654] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.681125 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6529a0cb514111b1bfd58537d255d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.657450 20038 tablet_service.cc:1511] Processing CreateTablet for tablet 43d7dcb7b21548889a850700415701a9 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260317 16:18:02.686141 20109 tablet_service.cc:1511] Processing CreateTablet for tablet 43d7dcb7b21548889a850700415701a9 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260317 16:18:02.695695 20109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d7dcb7b21548889a850700415701a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.695271 20256 tablet_service.cc:1511] Processing CreateTablet for tablet 062108d003c14795847b5db10bbabd0f (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260317 16:18:02.697357 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062108d003c14795847b5db10bbabd0f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:02.699340 20076 tablet.cc:2379] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 16:18:02.688505 20181 tablet_service.cc:1511] Processing CreateTablet for tablet 43d7dcb7b21548889a850700415701a9 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260317 16:18:02.693351 20321 tablet_bootstrap.cc:654] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.707175 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d7dcb7b21548889a850700415701a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.715292 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d7dcb7b21548889a850700415701a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.721335 20328 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:02.722039 20328 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:02.724169 20034 tablet_service.cc:1511] Processing CreateTablet for tablet 062108d003c14795847b5db10bbabd0f (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260317 16:18:02.725628 20034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062108d003c14795847b5db10bbabd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.725541 20328 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:02.760635 20318 tablet_bootstrap.cc:654] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:02.738404 20110 tablet_service.cc:1511] Processing CreateTablet for tablet 062108d003c14795847b5db10bbabd0f (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260317 16:18:02.772106 20320 tablet_bootstrap.cc:492] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:02.772751 20320 ts_tablet_manager.cc:1403] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.224s user 0.046s sys 0.020s
I20260317 16:18:02.776410 20320 raft_consensus.cc:359] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.777189 20320 raft_consensus.cc:385] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:02.777484 20320 raft_consensus.cc:740] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:02.783191 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 021523da58cc4c16babb94f5b4b5cee8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260317 16:18:02.781761 20182 tablet_service.cc:1511] Processing CreateTablet for tablet 021523da58cc4c16babb94f5b4b5cee8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260317 16:18:02.784605 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021523da58cc4c16babb94f5b4b5cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.785259 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021523da58cc4c16babb94f5b4b5cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.789417 20035 tablet_service.cc:1511] Processing CreateTablet for tablet 021523da58cc4c16babb94f5b4b5cee8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260317 16:18:02.792773 20113 tablet_service.cc:1511] Processing CreateTablet for tablet 1e0aed7c9dcd417d82dac37a8d010373 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260317 16:18:02.794232 20035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021523da58cc4c16babb94f5b4b5cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.790443 20185 tablet_service.cc:1511] Processing CreateTablet for tablet 1e0aed7c9dcd417d82dac37a8d010373 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260317 16:18:02.795897 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0aed7c9dcd417d82dac37a8d010373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.778368 20320 consensus_queue.cc:260] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.799268 20260 tablet_service.cc:1511] Processing CreateTablet for tablet 1e0aed7c9dcd417d82dac37a8d010373 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260317 16:18:02.800765 20260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0aed7c9dcd417d82dac37a8d010373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.801007 20320 ts_tablet_manager.cc:1434] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20260317 16:18:02.772104 20322 tablet_bootstrap.cc:492] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:02.801605 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062108d003c14795847b5db10bbabd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.815572 20327 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:02.819869 20329 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:02.829862 20109 tablet_service.cc:1511] Processing CreateTablet for tablet 951cd4af9fa84309b681fc8e9a6279ff (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
W20260317 16:18:02.826370 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:02.839885 20037 tablet_service.cc:1511] Processing CreateTablet for tablet 951cd4af9fa84309b681fc8e9a6279ff (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260317 16:18:02.860999 20327 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.852368 20322 ts_tablet_manager.cc:1403] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.264s user 0.015s sys 0.019s
I20260317 16:18:02.863076 20330 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:02.863729 20330 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:02.864812 20184 tablet_service.cc:1511] Processing CreateTablet for tablet 951cd4af9fa84309b681fc8e9a6279ff (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260317 16:18:02.865728 20322 raft_consensus.cc:359] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:02.866978 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0aed7c9dcd417d82dac37a8d010373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.861086 20329 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:02.869347 20184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 951cd4af9fa84309b681fc8e9a6279ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.870944 20330 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:02.866533 20322 raft_consensus.cc:385] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:02.890625 20322 raft_consensus.cc:740] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:02.895298 20321 tablet_bootstrap.cc:492] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:02.905004 20322 consensus_queue.cc:260] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:02.893072 20109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 951cd4af9fa84309b681fc8e9a6279ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:02.907632 20320 tablet_bootstrap.cc:492] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:02.893072 20327 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:02.919275 20321 ts_tablet_manager.cc:1403] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.325s user 0.025s sys 0.007s
I20260317 16:18:02.921891 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 951cd4af9fa84309b681fc8e9a6279ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.001348 20322 ts_tablet_manager.cc:1434] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.139s user 0.011s sys 0.000s
I20260317 16:18:02.986328 20259 tablet_service.cc:1511] Processing CreateTablet for tablet 7e5551ca32124ae686a779b9eafe976e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260317 16:18:03.015690 20329 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:03.016359 20038 tablet_service.cc:1511] Processing CreateTablet for tablet 7e5551ca32124ae686a779b9eafe976e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260317 16:18:03.030359 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5551ca32124ae686a779b9eafe976e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.033005 20259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5551ca32124ae686a779b9eafe976e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.004465 20186 tablet_service.cc:1511] Processing CreateTablet for tablet c902369cf6954cb695208719bf353017 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260317 16:18:02.992350 20108 tablet_service.cc:1511] Processing CreateTablet for tablet 7e5551ca32124ae686a779b9eafe976e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260317 16:18:03.090507 20033 tablet_service.cc:1511] Processing CreateTablet for tablet c902369cf6954cb695208719bf353017 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260317 16:18:03.092407 20033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c902369cf6954cb695208719bf353017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.108323 20318 tablet_bootstrap.cc:492] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:03.111284 20318 ts_tablet_manager.cc:1403] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.465s user 0.025s sys 0.029s
I20260317 16:18:03.111847 20107 tablet_service.cc:1511] Processing CreateTablet for tablet edd06ee443fc4680b217c7dcbc1f640a (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260317 16:18:03.100502 20320 tablet_bootstrap.cc:654] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:03.117504 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd06ee443fc4680b217c7dcbc1f640a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.121101 20256 tablet_service.cc:1511] Processing CreateTablet for tablet c902369cf6954cb695208719bf353017 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260317 16:18:03.126645 20321 raft_consensus.cc:359] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:03.127501 20321 raft_consensus.cc:385] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:03.127826 20321 raft_consensus.cc:740] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:03.128716 20321 consensus_queue.cc:260] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:03.130717 20322 tablet_bootstrap.cc:492] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:03.149485 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c902369cf6954cb695208719bf353017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.163787 20189 tablet_service.cc:1511] Processing CreateTablet for tablet edd06ee443fc4680b217c7dcbc1f640a (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260317 16:18:03.175168 20318 raft_consensus.cc:359] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.176072 20318 raft_consensus.cc:385] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:03.176448 20318 raft_consensus.cc:740] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:03.176687 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e5551ca32124ae686a779b9eafe976e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.177162 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:03.177376 20318 consensus_queue.cc:260] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.178867 20186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c902369cf6954cb695208719bf353017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.180024 20189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd06ee443fc4680b217c7dcbc1f640a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.178882 20125 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:03.182860 20165 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:03.188257 20327 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:03.188925 20327 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:03.189318 20327 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:03.200115 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:03.200882 20125 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:03.202704 20239 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:03.203797 20328 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:03.204178 20328 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:03.204501 20328 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:03.204725 20321 ts_tablet_manager.cc:1434] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.271s user 0.011s sys 0.000s
I20260317 16:18:03.206097 20321 tablet_bootstrap.cc:492] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:03.220396 20318 ts_tablet_manager.cc:1434] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.109s user 0.004s sys 0.005s
I20260317 16:18:03.231575 20318 tablet_bootstrap.cc:492] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:03.234407 20039 tablet_service.cc:1511] Processing CreateTablet for tablet 37f744805bd5488a8790301a2537fef1 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260317 16:18:03.235878 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f744805bd5488a8790301a2537fef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.236213 20255 tablet_service.cc:1511] Processing CreateTablet for tablet edd06ee443fc4680b217c7dcbc1f640a (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260317 16:18:03.238173 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd06ee443fc4680b217c7dcbc1f640a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.245095 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:03.247257 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:03.247123 20111 tablet_service.cc:1511] Processing CreateTablet for tablet 37f744805bd5488a8790301a2537fef1 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260317 16:18:03.248816 20111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f744805bd5488a8790301a2537fef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.244887 20327 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:03.260656 20236 leader_election.cc:343] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.267858 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:03.264982 20318 tablet_bootstrap.cc:654] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:03.270246 20125 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:03.277230 20088 leader_election.cc:343] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.274806 20327 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:03.315771 20327 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:03.316313 20327 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.317674 20321 tablet_bootstrap.cc:654] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:03.317312 20181 tablet_service.cc:1511] Processing CreateTablet for tablet 37f744805bd5488a8790301a2537fef1 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260317 16:18:03.319568 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f744805bd5488a8790301a2537fef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.320410 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:03.324558 20165 leader_election.cc:343] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.326365 20327 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:03.340585 20320 tablet_bootstrap.cc:492] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:03.341281 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:03.342486 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:03.366317 20050 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:03.343101 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:03.347141 20257 tablet_service.cc:1511] Processing CreateTablet for tablet 60cb13d3aec54fd58f4a4821692ab2b3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260317 16:18:03.352625 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:03.354884 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:03.377348 20199 raft_consensus.cc:2393] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:03.362043 20322 tablet_bootstrap.cc:654] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:03.372926 20162 leader_election.cc:343] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.394001 20257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60cb13d3aec54fd58f4a4821692ab2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.364117 20328 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:03.404950 20050 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:03.398597 20320 ts_tablet_manager.cc:1403] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.534s user 0.037s sys 0.040s
I20260317 16:18:03.384416 20162 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:03.420421 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 47674.4us
I20260317 16:18:03.423259 20352 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:03.422478 20256 tablet_service.cc:1511] Processing CreateTablet for tablet fd397002573f4a5ba2091161eaea1a47 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260317 16:18:03.414985 20187 tablet_service.cc:1511] Processing CreateTablet for tablet fd397002573f4a5ba2091161eaea1a47 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260317 16:18:03.427105 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd397002573f4a5ba2091161eaea1a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.427208 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd397002573f4a5ba2091161eaea1a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.428690 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:03.417369 20113 tablet_service.cc:1511] Processing CreateTablet for tablet 60cb13d3aec54fd58f4a4821692ab2b3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260317 16:18:03.430523 20125 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:03.448022 20038 tablet_service.cc:1511] Processing CreateTablet for tablet 60cb13d3aec54fd58f4a4821692ab2b3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260317 16:18:03.449493 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60cb13d3aec54fd58f4a4821692ab2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.449093 20016 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:03.450994 20329 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:03.441008 20162 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:03.452042 20273 raft_consensus.cc:2393] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:03.452329 20352 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:03.453279 20352 raft_consensus.cc:697] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:03.456751 20352 consensus_queue.cc:237] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.462024 20091 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:03.464321 20330 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:03.472438 20328 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:03.473551 20328 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:03.490368 20328 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.507195 20260 tablet_service.cc:1511] Processing CreateTablet for tablet d4f0e7842cc94d539a8537a2b131711b (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260317 16:18:03.507658 20328 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:03.508702 20260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f0e7842cc94d539a8537a2b131711b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.508495 20033 tablet_service.cc:1511] Processing CreateTablet for tablet 3324f667758c47ca8781d7f7b7822af8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260317 16:18:03.511276 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:03.512534 20033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3324f667758c47ca8781d7f7b7822af8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:03.512791 20236 leader_election.cc:343] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.513118 20182 tablet_service.cc:1511] Processing CreateTablet for tablet 3324f667758c47ca8781d7f7b7822af8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260317 16:18:03.514439 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60cb13d3aec54fd58f4a4821692ab2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.515489 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3324f667758c47ca8781d7f7b7822af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.523875 20321 tablet_bootstrap.cc:492] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:03.497470 20320 raft_consensus.cc:359] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.527468 20321 ts_tablet_manager.cc:1403] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.322s user 0.023s sys 0.026s
I20260317 16:18:03.528122 20320 raft_consensus.cc:385] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:03.528479 20320 raft_consensus.cc:740] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:03.529433 20320 consensus_queue.cc:260] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.531463 20321 raft_consensus.cc:359] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:03.532820 20321 raft_consensus.cc:385] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:03.533154 20321 raft_consensus.cc:740] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:03.534101 20321 consensus_queue.cc:260] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:03.540257 20037 tablet_service.cc:1511] Processing CreateTablet for tablet b7f7f224bb064944a05175b00d835fce (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260317 16:18:03.542062 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7f7f224bb064944a05175b00d835fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.491834 20110 tablet_service.cc:1511] Processing CreateTablet for tablet d4f0e7842cc94d539a8537a2b131711b (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260317 16:18:03.544394 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f0e7842cc94d539a8537a2b131711b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.494360 20185 tablet_service.cc:1511] Processing CreateTablet for tablet d4f0e7842cc94d539a8537a2b131711b (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260317 16:18:03.550940 20034 tablet_service.cc:1511] Processing CreateTablet for tablet a9f666d8f908427cb4523d57a45eb2b3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260317 16:18:03.551659 20320 ts_tablet_manager.cc:1434] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.114s user 0.005s sys 0.004s
I20260317 16:18:03.552289 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f0e7842cc94d539a8537a2b131711b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.552385 20034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f666d8f908427cb4523d57a45eb2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.552870 20320 tablet_bootstrap.cc:492] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:03.553086 20329 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:03.553635 20329 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:03.560439 20329 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:03.562189 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:03.502327 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:03.567775 20236 leader_election.cc:343] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.550940 20181 tablet_service.cc:1511] Processing CreateTablet for tablet a9f666d8f908427cb4523d57a45eb2b3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
W20260317 16:18:03.568432 20015 leader_election.cc:343] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.569303 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f666d8f908427cb4523d57a45eb2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.572613 20255 tablet_service.cc:1511] Processing CreateTablet for tablet b7f7f224bb064944a05175b00d835fce (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260317 16:18:03.574004 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7f7f224bb064944a05175b00d835fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.503140 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:03.578127 20259 tablet_service.cc:1511] Processing CreateTablet for tablet a9f666d8f908427cb4523d57a45eb2b3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260317 16:18:03.578733 20109 tablet_service.cc:1511] Processing CreateTablet for tablet e8fd5a883fbb4ae1bcc76cb626ccb7f2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260317 16:18:03.580271 20109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fd5a883fbb4ae1bcc76cb626ccb7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.581090 20111 tablet_service.cc:1511] Processing CreateTablet for tablet b7f7f224bb064944a05175b00d835fce (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260317 16:18:03.574071 20189 tablet_service.cc:1511] Processing CreateTablet for tablet e8fd5a883fbb4ae1bcc76cb626ccb7f2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260317 16:18:03.582664 20111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7f7f224bb064944a05175b00d835fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.582942 20035 tablet_service.cc:1511] Processing CreateTablet for tablet fd397002573f4a5ba2091161eaea1a47 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260317 16:18:03.584374 20035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd397002573f4a5ba2091161eaea1a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.579883 20259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f666d8f908427cb4523d57a45eb2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.582530 20108 tablet_service.cc:1511] Processing CreateTablet for tablet 3324f667758c47ca8781d7f7b7822af8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260317 16:18:03.589135 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3324f667758c47ca8781d7f7b7822af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.590971 20189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fd5a883fbb4ae1bcc76cb626ccb7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.602377 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:03.602521 20258 tablet_service.cc:1511] Processing CreateTablet for tablet e8fd5a883fbb4ae1bcc76cb626ccb7f2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260317 16:18:03.604851 20258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fd5a883fbb4ae1bcc76cb626ccb7f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:03.607905 20016 leader_election.cc:343] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.608316 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:03.608516 20016 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:03.609149 20199 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:03.609818 20329 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:03.611207 20238 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:03.614624 20328 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:03.615105 20328 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:03.615442 20328 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:03.616384 20322 tablet_bootstrap.cc:492] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:03.616941 20322 ts_tablet_manager.cc:1403] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.487s user 0.018s sys 0.009s
I20260317 16:18:03.620771 20318 tablet_bootstrap.cc:492] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:03.625933 20318 ts_tablet_manager.cc:1403] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.395s user 0.029s sys 0.058s
I20260317 16:18:03.644232 20039 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7d5e1825514e7abdeac801ec091d2c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260317 16:18:03.646059 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7d5e1825514e7abdeac801ec091d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.643464 20330 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:03.656246 20318 raft_consensus.cc:359] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.658061 20318 raft_consensus.cc:385] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:03.659003 20125 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:03.658439 20318 raft_consensus.cc:740] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:03.661489 20318 consensus_queue.cc:260] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.664822 20328 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.628861 20322 raft_consensus.cc:359] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.685122 20322 raft_consensus.cc:385] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:03.685499 20322 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:03.686049 20107 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7d5e1825514e7abdeac801ec091d2c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260317 16:18:03.689080 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7d5e1825514e7abdeac801ec091d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.691972 20318 ts_tablet_manager.cc:1434] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.066s user 0.010s sys 0.002s
I20260317 16:18:03.695585 20318 tablet_bootstrap.cc:492] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:03.687119 20321 ts_tablet_manager.cc:1434] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.159s user 0.005s sys 0.006s
I20260317 16:18:03.699870 20321 tablet_bootstrap.cc:492] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:03.706948 20318 tablet_bootstrap.cc:654] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:03.686419 20322 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.728550 20187 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7d5e1825514e7abdeac801ec091d2c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260317 16:18:03.730093 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7d5e1825514e7abdeac801ec091d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.739575 20322 ts_tablet_manager.cc:1434] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.122s user 0.012s sys 0.002s
I20260317 16:18:03.743837 20322 tablet_bootstrap.cc:492] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:03.745271 20239 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:03.747131 20355 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:03.751360 19948 catalog_manager.cc:5671] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:03.787202 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:03.787495 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:03.787916 20199 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:03.789224 20236 leader_election.cc:343] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.786170 20328 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:03.799739 20199 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:03.811692 20238 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:03.786113 20259 tablet_service.cc:1511] Processing CreateTablet for tablet b302990b671946b29b6e2fcf85413383 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260317 16:18:03.814455 20330 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:03.830896 20328 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:03.831472 20328 raft_consensus.cc:697] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:03.831255 20330 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:03.832062 20113 tablet_service.cc:1511] Processing CreateTablet for tablet b302990b671946b29b6e2fcf85413383 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260317 16:18:03.833467 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b302990b671946b29b6e2fcf85413383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.836787 20259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b302990b671946b29b6e2fcf85413383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.837731 20181 tablet_service.cc:1511] Processing CreateTablet for tablet fb492ec38d074e94994a3a2a9e245bac (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260317 16:18:03.812840 20355 raft_consensus.cc:697] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:03.841918 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb492ec38d074e94994a3a2a9e245bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.846833 20033 tablet_service.cc:1511] Processing CreateTablet for tablet fb492ec38d074e94994a3a2a9e245bac (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260317 16:18:03.848349 20033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb492ec38d074e94994a3a2a9e245bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.851668 20330 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:03.855630 20112 tablet_service.cc:1511] Processing CreateTablet for tablet ead98604210744a2b26b15fb54b5ea29 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260317 16:18:03.832330 20328 consensus_queue.cc:237] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:03.863574 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:03.865322 20088 leader_election.cc:343] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:03.865777 20182 tablet_service.cc:1511] Processing CreateTablet for tablet ead98604210744a2b26b15fb54b5ea29 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260317 16:18:03.872876 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead98604210744a2b26b15fb54b5ea29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.872587 20257 tablet_service.cc:1511] Processing CreateTablet for tablet fb492ec38d074e94994a3a2a9e245bac (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260317 16:18:03.873994 20257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb492ec38d074e94994a3a2a9e245bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.880946 20112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead98604210744a2b26b15fb54b5ea29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.881094 20258 tablet_service.cc:1511] Processing CreateTablet for tablet ead98604210744a2b26b15fb54b5ea29 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260317 16:18:03.894696 20329 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:03.895489 20258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead98604210744a2b26b15fb54b5ea29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.895311 20329 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:03.897933 20329 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:03.901086 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:03.901803 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:03.904767 20016 leader_election.cc:343] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:03.914366 20016 leader_election.cc:343] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:03.914842 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:03.914916 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:03.917482 20329 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:03.923274 20355 consensus_queue.cc:237] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:03.933043 20320 tablet_bootstrap.cc:700] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent opening tablet: real 0.134s user 0.002s sys 0.001s
I20260317 16:18:03.879338 20037 tablet_service.cc:1511] Processing CreateTablet for tablet b302990b671946b29b6e2fcf85413383 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260317 16:18:03.958695 20034 tablet_service.cc:1511] Processing CreateTablet for tablet 83e3771d4d494b8caa5d647d8073666c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260317 16:18:03.961196 20318 tablet_bootstrap.cc:492] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:03.961848 20318 ts_tablet_manager.cc:1403] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.267s user 0.027s sys 0.017s
I20260317 16:18:03.976073 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 9a97b610497d410fb77dc8d4e5aca57e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260317 16:18:03.977632 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a97b610497d410fb77dc8d4e5aca57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.977713 20038 tablet_service.cc:1511] Processing CreateTablet for tablet 9a97b610497d410fb77dc8d4e5aca57e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260317 16:18:03.979858 20322 tablet_bootstrap.cc:654] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:03.968984 20321 tablet_bootstrap.cc:654] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:03.988880 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b302990b671946b29b6e2fcf85413383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:03.971194 20320 tablet_bootstrap.cc:654] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.005467 20260 tablet_service.cc:1511] Processing CreateTablet for tablet d8bd34d6b1ae4fe5bf58a857d0f837ea (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260317 16:18:04.007803 20260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bd34d6b1ae4fe5bf58a857d0f837ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.008906 20318 raft_consensus.cc:359] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.010210 20318 raft_consensus.cc:385] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:04.012497 20108 tablet_service.cc:1511] Processing CreateTablet for tablet 83e3771d4d494b8caa5d647d8073666c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260317 16:18:04.019059 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e3771d4d494b8caa5d647d8073666c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.029480 20040 tablet_service.cc:1511] Processing CreateTablet for tablet d8bd34d6b1ae4fe5bf58a857d0f837ea (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260317 16:18:03.998427 20189 tablet_service.cc:1511] Processing CreateTablet for tablet 83e3771d4d494b8caa5d647d8073666c (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260317 16:18:04.010528 20318 raft_consensus.cc:740] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:04.049008 20352 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 16:18:04.060321 20185 tablet_service.cc:1511] Processing CreateTablet for tablet d8bd34d6b1ae4fe5bf58a857d0f837ea (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260317 16:18:04.065232 20352 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:04.068018 19947 catalog_manager.cc:5671] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:04.079397 20189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e3771d4d494b8caa5d647d8073666c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.088332 20109 tablet_service.cc:1511] Processing CreateTablet for tablet 9a97b610497d410fb77dc8d4e5aca57e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260317 16:18:04.089919 20109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a97b610497d410fb77dc8d4e5aca57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.096105 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:04.098786 20090 leader_election.cc:343] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.099346 20090 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:04.119932 20110 tablet_service.cc:1511] Processing CreateTablet for tablet 0709d1e3fc7349f2b93ed3e602c1a723 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260317 16:18:04.121222 20256 tablet_service.cc:1511] Processing CreateTablet for tablet 0709d1e3fc7349f2b93ed3e602c1a723 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260317 16:18:04.122619 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0709d1e3fc7349f2b93ed3e602c1a723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.123765 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a97b610497d410fb77dc8d4e5aca57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.113029 20330 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:04.121640 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0709d1e3fc7349f2b93ed3e602c1a723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.131242 20321 tablet_bootstrap.cc:492] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:04.126232 20187 tablet_service.cc:1511] Processing CreateTablet for tablet 0709d1e3fc7349f2b93ed3e602c1a723 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260317 16:18:04.131505 20355 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:04.140012 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bd34d6b1ae4fe5bf58a857d0f837ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.154564 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0709d1e3fc7349f2b93ed3e602c1a723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.155007 20039 tablet_service.cc:1511] Processing CreateTablet for tablet 80b48fd655d14d649c65e99261b01ca9 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260317 16:18:04.151338 20355 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.161687 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:04.166182 20040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bd34d6b1ae4fe5bf58a857d0f837ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:04.167759 20239 leader_election.cc:343] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.169495 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:04.171120 20034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e3771d4d494b8caa5d647d8073666c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:04.171037 20238 leader_election.cc:343] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.171741 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:04.172567 20328 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:04.175426 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b48fd655d14d649c65e99261b01ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.151718 20321 ts_tablet_manager.cc:1403] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.452s user 0.040s sys 0.070s
I20260317 16:18:04.103375 20318 consensus_queue.cc:260] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.193621 20322 tablet_bootstrap.cc:492] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:04.194278 20322 ts_tablet_manager.cc:1403] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.451s user 0.008s sys 0.022s
I20260317 16:18:04.203233 20321 raft_consensus.cc:359] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.204241 20321 raft_consensus.cc:385] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:04.205035 20321 raft_consensus.cc:740] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:04.210323 20322 raft_consensus.cc:359] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.211194 20322 raft_consensus.cc:385] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:04.211527 20322 raft_consensus.cc:740] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:04.212417 20322 consensus_queue.cc:260] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.216044 20322 ts_tablet_manager.cc:1434] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.021s user 0.004s sys 0.004s
I20260317 16:18:04.162581 20355 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:04.219190 20318 ts_tablet_manager.cc:1434] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.257s user 0.013s sys 0.003s
I20260317 16:18:04.220768 20318 tablet_bootstrap.cc:492] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:04.226606 20320 tablet_bootstrap.cc:492] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:04.232797 20322 tablet_bootstrap.cc:492] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:04.206362 20321 consensus_queue.cc:260] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.240079 20355 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:04.242132 20355 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.240168 20112 tablet_service.cc:1511] Processing CreateTablet for tablet 80b48fd655d14d649c65e99261b01ca9 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260317 16:18:04.231649 20320 ts_tablet_manager.cc:1403] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.678s user 0.018s sys 0.015s
I20260317 16:18:04.258617 20186 tablet_service.cc:1511] Processing CreateTablet for tablet 80b48fd655d14d649c65e99261b01ca9 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260317 16:18:04.259116 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:04.260111 20186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b48fd655d14d649c65e99261b01ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.261749 20321 ts_tablet_manager.cc:1434] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.084s user 0.010s sys 0.003s
I20260317 16:18:04.267302 20321 tablet_bootstrap.cc:492] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:04.266300 20330 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:04.272918 20239 leader_election.cc:343] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.273368 20330 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.281316 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:04.284493 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:04.285514 20330 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:04.287587 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:04.288616 20238 leader_election.cc:343] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.289099 20238 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:04.290232 20328 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:04.291097 20091 leader_election.cc:343] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.291834 20037 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff46c37d04d446981ad8728f73c6e1f (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260317 16:18:04.293303 20033 tablet_service.cc:1511] Processing CreateTablet for tablet b444cc548a6940e98033a9a247164c51 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260317 16:18:04.295207 20033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b444cc548a6940e98033a9a247164c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.299935 20182 tablet_service.cc:1511] Processing CreateTablet for tablet b444cc548a6940e98033a9a247164c51 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260317 16:18:04.282334 20049 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:04.305135 20088 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:04.307279 20330 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:04.307823 20330 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:04.308305 20330 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:04.314810 20049 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:04.319074 20328 consensus_queue.cc:1048] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:04.323247 20112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b48fd655d14d649c65e99261b01ca9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:04.323791 19935 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 54907.4us
I20260317 16:18:04.326437 20322 tablet_bootstrap.cc:654] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.328469 20110 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff46c37d04d446981ad8728f73c6e1f (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260317 16:18:04.343928 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff46c37d04d446981ad8728f73c6e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.345691 20320 raft_consensus.cc:359] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:04.346710 20320 raft_consensus.cc:385] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:04.292951 20355 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:04.355445 20236 consensus_peers.cc:597] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 -> Peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Couldn't send request to peer 9a3afb098d6a45f0b405566771a1157b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:18:04.362881 20328 consensus_queue.cc:1048] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:04.347055 20320 raft_consensus.cc:740] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:04.370981 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b444cc548a6940e98033a9a247164c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.370545 20320 consensus_queue.cc:260] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:04.379421 20113 tablet_service.cc:1511] Processing CreateTablet for tablet aa7ec7cb7aad43e8950899557d4022f7 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260317 16:18:04.383539 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa7ec7cb7aad43e8950899557d4022f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.408879 20259 tablet_service.cc:1511] Processing CreateTablet for tablet aa7ec7cb7aad43e8950899557d4022f7 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260317 16:18:04.373989 20184 tablet_service.cc:1511] Processing CreateTablet for tablet aa7ec7cb7aad43e8950899557d4022f7 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260317 16:18:04.441079 20039 tablet_service.cc:1511] Processing CreateTablet for tablet d3bc11389a01493f9af4639ba9d9e449 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260317 16:18:04.383417 20256 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff46c37d04d446981ad8728f73c6e1f (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260317 16:18:04.430742 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff46c37d04d446981ad8728f73c6e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.474463 20034 tablet_service.cc:1511] Processing CreateTablet for tablet 4dfc403ac89c4d339adb26bfd789d1a2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260317 16:18:04.481063 20259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa7ec7cb7aad43e8950899557d4022f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.495214 20377 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:04.497081 20184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa7ec7cb7aad43e8950899557d4022f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.507215 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bc11389a01493f9af4639ba9d9e449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.476044 20107 tablet_service.cc:1511] Processing CreateTablet for tablet 4dfc403ac89c4d339adb26bfd789d1a2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260317 16:18:04.542851 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dfc403ac89c4d339adb26bfd789d1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.547730 20330 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.548333 20038 tablet_service.cc:1511] Processing CreateTablet for tablet d7faa93fffc94522a88d5c47721186bd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260317 16:18:04.548707 20187 tablet_service.cc:1511] Processing CreateTablet for tablet fca192bf1e8848e2a2dac227f537c9e2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260317 16:18:04.449762 20108 tablet_service.cc:1511] Processing CreateTablet for tablet d3bc11389a01493f9af4639ba9d9e449 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260317 16:18:04.542898 20257 tablet_service.cc:1511] Processing CreateTablet for tablet 4dfc403ac89c4d339adb26bfd789d1a2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260317 16:18:04.552558 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bc11389a01493f9af4639ba9d9e449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.552820 20255 tablet_service.cc:1511] Processing CreateTablet for tablet d7faa93fffc94522a88d5c47721186bd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260317 16:18:04.447528 20260 tablet_service.cc:1511] Processing CreateTablet for tablet b444cc548a6940e98033a9a247164c51 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260317 16:18:04.508029 20318 tablet_bootstrap.cc:700] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Time spent opening tablet: real 0.122s user 0.003s sys 0.000s
I20260317 16:18:04.565115 20109 tablet_service.cc:1511] Processing CreateTablet for tablet fca192bf1e8848e2a2dac227f537c9e2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260317 16:18:04.566795 20318 tablet_bootstrap.cc:654] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.567396 20109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca192bf1e8848e2a2dac227f537c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.566438 19947 catalog_manager.cc:5671] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:04.478111 20181 tablet_service.cc:1511] Processing CreateTablet for tablet d3bc11389a01493f9af4639ba9d9e449 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260317 16:18:04.533653 20377 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:04.536283 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff46c37d04d446981ad8728f73c6e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.516914 20320 ts_tablet_manager.cc:1434] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.266s user 0.023s sys 0.007s
I20260317 16:18:04.592173 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7faa93fffc94522a88d5c47721186bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.596017 20328 consensus_queue.cc:1048] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:04.571310 20321 tablet_bootstrap.cc:654] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.594071 20258 tablet_service.cc:1511] Processing CreateTablet for tablet fca192bf1e8848e2a2dac227f537c9e2 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260317 16:18:04.477684 20189 tablet_service.cc:1511] Processing CreateTablet for tablet d7faa93fffc94522a88d5c47721186bd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260317 16:18:04.607682 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca192bf1e8848e2a2dac227f537c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.600179 20329 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:04.619436 20329 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.628316 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:04.629051 20377 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:04.616715 20034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dfc403ac89c4d339adb26bfd789d1a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:04.646209 20091 leader_election.cc:343] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.647266 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:04.649842 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:04.650520 20124 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
W20260317 16:18:04.651713 20162 leader_election.cc:343] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.634490 20330 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:04.652390 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:04.652891 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3bc11389a01493f9af4639ba9d9e449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.653627 20352 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:04.654228 20352 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:04.654605 20352 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:04.658025 20381 log.cc:927] Time spent T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260317 16:18:04.659993 20189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7faa93fffc94522a88d5c47721186bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.663327 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:04.664364 20048 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:04.639918 20260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b444cc548a6940e98033a9a247164c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.685653 20257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dfc403ac89c4d339adb26bfd789d1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.687690 20258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca192bf1e8848e2a2dac227f537c9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.690438 20322 tablet_bootstrap.cc:492] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:04.691231 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7faa93fffc94522a88d5c47721186bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:04.684428 20378 log.cc:927] Time spent T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260317 16:18:04.691387 20322 ts_tablet_manager.cc:1403] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.459s user 0.020s sys 0.007s
I20260317 16:18:04.719514 20040 tablet_service.cc:1511] Processing CreateTablet for tablet c963e7aa6d46472691e7e0de834008ab (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260317 16:18:04.722303 20040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c963e7aa6d46472691e7e0de834008ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.730979 20320 tablet_bootstrap.cc:492] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:04.777055 20329 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:04.778407 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:04.778638 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:04.782199 20016 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:04.800065 20329 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:04.826416 20185 tablet_service.cc:1511] Processing CreateTablet for tablet c963e7aa6d46472691e7e0de834008ab (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260317 16:18:04.826797 20113 tablet_service.cc:1511] Processing CreateTablet for tablet c963e7aa6d46472691e7e0de834008ab (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260317 16:18:04.829852 20113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c963e7aa6d46472691e7e0de834008ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.829306 20322 raft_consensus.cc:359] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.830317 20322 raft_consensus.cc:385] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:04.830619 20322 raft_consensus.cc:740] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:04.831214 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c963e7aa6d46472691e7e0de834008ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.832075 20048 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:04.833954 20088 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:04.835179 20330 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:04.835723 20330 raft_consensus.cc:697] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:04.836726 20330 consensus_queue.cc:237] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.839063 20322 consensus_queue.cc:260] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:04.854061 20322 ts_tablet_manager.cc:1434] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.149s user 0.006s sys 0.005s
I20260317 16:18:04.855301 20322 tablet_bootstrap.cc:492] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:04.862730 20321 tablet_bootstrap.cc:492] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:04.863350 20321 ts_tablet_manager.cc:1403] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.601s user 0.018s sys 0.016s
I20260317 16:18:04.867630 20263 tablet_service.cc:1511] Processing CreateTablet for tablet f26f14718ce8437ab76d99156bfa1fa3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260317 16:18:04.872188 20263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f26f14718ce8437ab76d99156bfa1fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.872471 20322 tablet_bootstrap.cc:654] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.866513 20321 raft_consensus.cc:359] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:04.880750 20321 raft_consensus.cc:385] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:04.881116 20321 raft_consensus.cc:740] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:04.883572 20038 tablet_service.cc:1511] Processing CreateTablet for tablet f26f14718ce8437ab76d99156bfa1fa3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260317 16:18:04.848222 20318 tablet_bootstrap.cc:492] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:04.812107 20352 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:04.888507 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f26f14718ce8437ab76d99156bfa1fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.907421 20187 tablet_service.cc:1511] Processing CreateTablet for tablet 3a2894e5c7f64210ba697507a2823f16 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260317 16:18:04.908936 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2894e5c7f64210ba697507a2823f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.913179 20033 tablet_service.cc:1511] Processing CreateTablet for tablet 3a2894e5c7f64210ba697507a2823f16 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260317 16:18:04.914796 20033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2894e5c7f64210ba697507a2823f16. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:04.920604 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:04.921171 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 3a2894e5c7f64210ba697507a2823f16 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260317 16:18:04.921077 20110 tablet_service.cc:1511] Processing CreateTablet for tablet f26f14718ce8437ab76d99156bfa1fa3 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260317 16:18:04.891068 20321 consensus_queue.cc:260] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:04.924886 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f26f14718ce8437ab76d99156bfa1fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.925710 20321 ts_tablet_manager.cc:1434] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.062s user 0.003s sys 0.005s
I20260317 16:18:04.931587 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2894e5c7f64210ba697507a2823f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.937407 20111 tablet_service.cc:1511] Processing CreateTablet for tablet 091c367bf7d84c61b2a329d98d708cc8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260317 16:18:04.939073 20111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091c367bf7d84c61b2a329d98d708cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.940690 20320 tablet_bootstrap.cc:654] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.942894 20318 ts_tablet_manager.cc:1403] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.722s user 0.035s sys 0.012s
I20260317 16:18:04.945711 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:04.946414 20125 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:04.949455 20321 tablet_bootstrap.cc:492] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:04.945400 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:04.951630 20182 tablet_service.cc:1511] Processing CreateTablet for tablet 091c367bf7d84c61b2a329d98d708cc8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260317 16:18:04.953444 20162 leader_election.cc:343] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:04.962893 20261 tablet_service.cc:1511] Processing CreateTablet for tablet 091c367bf7d84c61b2a329d98d708cc8 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260317 16:18:04.965655 20261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091c367bf7d84c61b2a329d98d708cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.971073 20352 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:04.973999 20125 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:04.975756 20321 tablet_bootstrap.cc:654] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:04.945984 20318 raft_consensus.cc:359] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:04.970980 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091c367bf7d84c61b2a329d98d708cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:04.986136 20039 tablet_service.cc:1511] Processing CreateTablet for tablet fb3cfd29098d4ab9a179a3d07ebb75ee (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260317 16:18:04.990275 20318 raft_consensus.cc:385] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:18:04.990293 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 66137.4us
I20260317 16:18:05.025712 20039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3cfd29098d4ab9a179a3d07ebb75ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.031984 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:05.032645 20108 tablet_service.cc:1511] Processing CreateTablet for tablet fb3cfd29098d4ab9a179a3d07ebb75ee (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260317 16:18:05.034025 20108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3cfd29098d4ab9a179a3d07ebb75ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.035209 20352 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:05.035789 20352 raft_consensus.cc:697] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:05.036302 20322 tablet_bootstrap.cc:492] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:05.037048 20322 ts_tablet_manager.cc:1403] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.182s user 0.013s sys 0.004s
I20260317 16:18:05.037106 20352 consensus_queue.cc:237] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.046937 20187 tablet_service.cc:1511] Processing CreateTablet for tablet fb3cfd29098d4ab9a179a3d07ebb75ee (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260317 16:18:05.048306 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3cfd29098d4ab9a179a3d07ebb75ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.070637 19947 catalog_manager.cc:5671] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:05.018198 20318 raft_consensus.cc:740] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.089085 20322 raft_consensus.cc:359] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.090175 20322 raft_consensus.cc:385] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.090488 20322 raft_consensus.cc:740] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.095682 20037 tablet_service.cc:1511] Processing CreateTablet for tablet 81baa6e6b5864f69b8da5c3418aea90a (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260317 16:18:05.098680 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81baa6e6b5864f69b8da5c3418aea90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.103157 20318 consensus_queue.cc:260] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.106698 20318 ts_tablet_manager.cc:1434] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.163s user 0.006s sys 0.003s
I20260317 16:18:05.116953 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 81baa6e6b5864f69b8da5c3418aea90a (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260317 16:18:05.118556 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81baa6e6b5864f69b8da5c3418aea90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.136813 19946 catalog_manager.cc:5671] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:05.147125 20322 consensus_queue.cc:260] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.153151 20322 ts_tablet_manager.cc:1434] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.116s user 0.005s sys 0.005s
I20260317 16:18:05.177323 20322 tablet_bootstrap.cc:492] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:05.186192 20110 tablet_service.cc:1511] Processing CreateTablet for tablet 81baa6e6b5864f69b8da5c3418aea90a (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260317 16:18:05.187700 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81baa6e6b5864f69b8da5c3418aea90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.188557 20318 tablet_bootstrap.cc:492] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:05.190297 20321 tablet_bootstrap.cc:492] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:05.190904 20321 ts_tablet_manager.cc:1403] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.242s user 0.027s sys 0.032s
I20260317 16:18:05.191069 20392 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.191598 20392 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:05.194038 20321 raft_consensus.cc:359] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.194872 20321 raft_consensus.cc:385] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.194949 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:05.195254 20321 raft_consensus.cc:740] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.195814 20049 raft_consensus.cc:2393] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:05.197245 20187 tablet_service.cc:1511] Processing CreateTablet for tablet fd3be07aef26463ba2eaa9c72c6fa094 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260317 16:18:05.197961 20392 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:05.198794 20187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3be07aef26463ba2eaa9c72c6fa094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.199476 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.200436 20258 tablet_service.cc:1511] Processing CreateTablet for tablet fd3be07aef26463ba2eaa9c72c6fa094 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260317 16:18:05.201992 20258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3be07aef26463ba2eaa9c72c6fa094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.201848 20239 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:05.197110 20321 consensus_queue.cc:260] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.202212 20107 tablet_service.cc:1511] Processing CreateTablet for tablet 34a510b634dc4a808c9f7af78ba4cd7e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260317 16:18:05.197801 20038 tablet_service.cc:1511] Processing CreateTablet for tablet fd3be07aef26463ba2eaa9c72c6fa094 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260317 16:18:05.207214 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a510b634dc4a808c9f7af78ba4cd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.208027 20322 tablet_bootstrap.cc:654] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.218922 20321 ts_tablet_manager.cc:1434] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.028s user 0.008s sys 0.001s
I20260317 16:18:05.222075 20391 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.222297 20320 tablet_bootstrap.cc:492] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:05.222668 20391 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.208732 20392 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.231206 20391 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:05.232489 20185 tablet_service.cc:1511] Processing CreateTablet for tablet 34a510b634dc4a808c9f7af78ba4cd7e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260317 16:18:05.234043 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a510b634dc4a808c9f7af78ba4cd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.234071 20329 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.235379 20329 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.238374 20329 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:05.241858 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:05.243474 20015 leader_election.cc:343] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.243623 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:05.241884 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:05.244755 20049 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:05.246090 20236 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:05.247797 20016 leader_election.cc:343] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.248514 20016 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:05.249972 20329 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:05.250533 20318 tablet_bootstrap.cc:654] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.251535 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:05.251019 20391 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:05.243327 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 34a510b634dc4a808c9f7af78ba4cd7e (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260317 16:18:05.260196 20392 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:05.265452 20321 tablet_bootstrap.cc:492] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:05.272243 20038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3be07aef26463ba2eaa9c72c6fa094. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 16:18:05.278079 20238 leader_election.cc:343] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.239208 20320 ts_tablet_manager.cc:1403] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.543s user 0.028s sys 0.016s
I20260317 16:18:05.276068 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a510b634dc4a808c9f7af78ba4cd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.275323 20391 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:05.282359 20391 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.287084 20037 tablet_service.cc:1511] Processing CreateTablet for tablet 65896e94a12d4953b9bbfff15ac73867 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260317 16:18:05.288594 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65896e94a12d4953b9bbfff15ac73867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.303279 20185 tablet_service.cc:1511] Processing CreateTablet for tablet 65896e94a12d4953b9bbfff15ac73867 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260317 16:18:05.306293 20185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65896e94a12d4953b9bbfff15ac73867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.308166 20256 tablet_service.cc:1511] Processing CreateTablet for tablet c198f5201a944b26bd4d2f870953e117 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260317 16:18:05.309715 20256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c198f5201a944b26bd4d2f870953e117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.310139 20321 tablet_bootstrap.cc:654] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.321091 20320 raft_consensus.cc:359] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.321899 20320 raft_consensus.cc:385] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.322199 20320 raft_consensus.cc:740] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.322502 20322 tablet_bootstrap.cc:492] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:05.323467 20322 ts_tablet_manager.cc:1403] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.146s user 0.013s sys 0.005s
I20260317 16:18:05.333644 20034 tablet_service.cc:1511] Processing CreateTablet for tablet c198f5201a944b26bd4d2f870953e117 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260317 16:18:05.335846 20107 tablet_service.cc:1511] Processing CreateTablet for tablet 65896e94a12d4953b9bbfff15ac73867 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260317 16:18:05.334131 20318 tablet_bootstrap.cc:492] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:05.338593 20107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65896e94a12d4953b9bbfff15ac73867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.339097 20034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c198f5201a944b26bd4d2f870953e117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.338690 20318 ts_tablet_manager.cc:1403] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.003s
I20260317 16:18:05.340420 20320 consensus_queue.cc:260] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.347690 20318 raft_consensus.cc:359] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.348553 20318 raft_consensus.cc:385] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.348914 20318 raft_consensus.cc:740] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.328352 20391 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.350479 20322 raft_consensus.cc:359] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:05.351711 20322 raft_consensus.cc:385] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.352079 20322 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.352373 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:05.352669 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:05.353580 20199 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.358639 20110 tablet_service.cc:1511] Processing CreateTablet for tablet c198f5201a944b26bd4d2f870953e117 (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260317 16:18:05.360743 20110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c198f5201a944b26bd4d2f870953e117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.353086 20049 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.362888 20391 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:05.365374 20320 ts_tablet_manager.cc:1434] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.074s user 0.007s sys 0.000s
I20260317 16:18:05.366132 20322 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:05.367332 20320 tablet_bootstrap.cc:492] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:05.373412 20037 tablet_service.cc:1511] Processing CreateTablet for tablet 7ef41a8cfc184979b0e3fc5d3b370bcd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260317 16:18:05.377810 20037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef41a8cfc184979b0e3fc5d3b370bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.369187 20322 ts_tablet_manager.cc:1434] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.045s user 0.005s sys 0.002s
I20260317 16:18:05.370162 20182 tablet_service.cc:1511] Processing CreateTablet for tablet 7ef41a8cfc184979b0e3fc5d3b370bcd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260317 16:18:05.392236 20182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef41a8cfc184979b0e3fc5d3b370bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.392302 20322 tablet_bootstrap.cc:492] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:05.369607 20255 tablet_service.cc:1511] Processing CreateTablet for tablet 7ef41a8cfc184979b0e3fc5d3b370bcd (DEFAULT_TABLE table=test-workload [id=b34906c5f0e84d5b916910674fc7e084]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260317 16:18:05.378499 20352 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.393615 20352 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.382143 20321 tablet_bootstrap.cc:492] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:05.394749 20321 ts_tablet_manager.cc:1403] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.005s
I20260317 16:18:05.395908 20352 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:05.398989 20321 raft_consensus.cc:359] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.399556 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:05.387470 20320 tablet_bootstrap.cc:654] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.402859 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef41a8cfc184979b0e3fc5d3b370bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 16:18:05.401612 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.400099 20321 raft_consensus.cc:385] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.414152 20321 raft_consensus.cc:740] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.414978 20321 consensus_queue.cc:260] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:05.416303 20162 leader_election.cc:343] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.417865 20321 ts_tablet_manager.cc:1434] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20260317 16:18:05.420601 20199 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:05.426215 20238 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:05.426970 20318 consensus_queue.cc:260] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.428530 20330 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.427150 20391 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:05.429638 20330 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.430365 20391 raft_consensus.cc:697] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:05.431165 20125 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:05.427052 20321 tablet_bootstrap.cc:492] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:05.433933 20391 consensus_queue.cc:237] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.435761 20049 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:05.438151 20165 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:05.439867 20352 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:05.440397 20352 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:05.440834 20352 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.449412 20321 tablet_bootstrap.cc:654] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.451786 20330 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:05.455273 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:05.443222 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:05.470693 20330 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.471994 20330 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.474542 20322 tablet_bootstrap.cc:654] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:05.473680 20088 leader_election.cc:343] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.488314 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:05.490070 20090 leader_election.cc:343] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.490386 20330 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:05.491139 20330 consensus_queue.cc:1048] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:05.473789 20352 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.491647 20320 tablet_bootstrap.cc:492] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:05.496016 20320 ts_tablet_manager.cc:1403] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.129s user 0.035s sys 0.000s
I20260317 16:18:05.502501 20199 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:05.510452 20090 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:05.512722 20403 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.513592 20048 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.507678 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:05.527216 20329 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.533420 20329 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.525235 20273 raft_consensus.cc:2468] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:05.543349 20091 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:05.546808 19946 catalog_manager.cc:5671] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:05.548915 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:05.549742 20320 raft_consensus.cc:359] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:05.550894 20015 leader_election.cc:343] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.551295 20352 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:05.548677 20329 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:05.552772 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:05.554013 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:05.554157 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.555130 20318 ts_tablet_manager.cc:1434] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.211s user 0.021s sys 0.015s
I20260317 16:18:05.557086 20318 tablet_bootstrap.cc:492] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
W20260317 16:18:05.557233 20162 leader_election.cc:343] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.550575 20320 raft_consensus.cc:385] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.563158 20402 consensus_queue.cc:1048] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260317 16:18:05.567577 20322 tablet_bootstrap.cc:492] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:05.568118 20322 ts_tablet_manager.cc:1403] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.176s user 0.017s sys 0.014s
I20260317 16:18:05.570245 20322 raft_consensus.cc:359] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:05.558142 20016 leader_election.cc:343] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.570742 20322 raft_consensus.cc:385] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.571074 20016 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:05.571503 20322 raft_consensus.cc:740] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.572098 20329 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:05.572212 20322 consensus_queue.cc:260] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.573777 20318 tablet_bootstrap.cc:654] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.574414 20322 ts_tablet_manager.cc:1434] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260317 16:18:05.575834 20322 tablet_bootstrap.cc:492] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:05.559346 20330 raft_consensus.cc:2804] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:05.589936 20330 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:05.590407 20330 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.559369 20320 raft_consensus.cc:740] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.594866 20322 tablet_bootstrap.cc:654] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.601009 20321 tablet_bootstrap.cc:492] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:05.602195 20321 ts_tablet_manager.cc:1403] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.175s user 0.017s sys 0.001s
I20260317 16:18:05.607259 20391 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.608330 20391 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:05.611193 20391 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:05.614019 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.594847 20320 consensus_queue.cc:260] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:05.624384 20239 leader_election.cc:343] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.630434 20318 tablet_bootstrap.cc:492] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:05.631108 20318 ts_tablet_manager.cc:1403] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.075s user 0.020s sys 0.006s
I20260317 16:18:05.635030 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:05.635349 20330 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.651044 20330 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:05.640376 20352 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:05.652136 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:05.646358 20408 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.646034 20321 raft_consensus.cc:359] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.654469 20321 raft_consensus.cc:385] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.654922 20321 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.664853 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:05.667770 20408 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.670737 20320 ts_tablet_manager.cc:1434] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.174s user 0.012s sys 0.000s
I20260317 16:18:05.673161 20321 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.672192 20320 tablet_bootstrap.cc:492] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:05.678079 20272 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:05.685350 20236 leader_election.cc:343] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.703388 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:05.706311 20408 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:05.708277 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.709398 20123 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:05.710876 20016 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:05.711149 20403 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:05.693394 20090 leader_election.cc:343] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.718900 20236 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:05.721809 20391 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:05.710472 20318 raft_consensus.cc:359] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:05.722349 20318 raft_consensus.cc:385] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.722668 20318 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.723733 20318 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:05.726790 20352 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:05.728341 20124 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:05.730552 20320 tablet_bootstrap.cc:654] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.731521 20165 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:05.733336 20272 raft_consensus.cc:2468] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:05.751824 20322 tablet_bootstrap.cc:492] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:05.752753 20322 ts_tablet_manager.cc:1403] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.177s user 0.038s sys 0.036s
I20260317 16:18:05.745901 20352 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:05.758916 20352 raft_consensus.cc:697] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:05.761042 20408 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:05.769870 20408 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:05.770431 20408 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:05.770920 20391 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.772804 20391 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.787184 20091 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:05.787941 20322 raft_consensus.cc:359] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.788357 20402 raft_consensus.cc:2804] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:05.788733 20322 raft_consensus.cc:385] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:05.789203 20322 raft_consensus.cc:740] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.790266 20322 consensus_queue.cc:260] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.793823 20402 raft_consensus.cc:697] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:05.762064 20352 consensus_queue.cc:237] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.814260 20321 ts_tablet_manager.cc:1434] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.212s user 0.017s sys 0.034s
I20260317 16:18:05.830044 20322 ts_tablet_manager.cc:1434] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.077s user 0.015s sys 0.004s
I20260317 16:18:05.831300 20322 tablet_bootstrap.cc:492] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:05.837495 20321 tablet_bootstrap.cc:492] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:05.830210 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 52909.7us
I20260317 16:18:05.867838 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.859683 20402 consensus_queue.cc:237] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:05.869639 20239 leader_election.cc:343] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.872277 20391 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:05.865468 20408 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.876777 20318 ts_tablet_manager.cc:1434] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.245s user 0.008s sys 0.004s
I20260317 16:18:05.878901 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:05.879724 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:05.878918 20049 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:05.887349 20238 leader_election.cc:343] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.888113 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:05.891863 20321 tablet_bootstrap.cc:654] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.892314 20048 raft_consensus.cc:1240] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:05.895503 20420 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:05.899514 20391 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.896251 20392 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:05.900347 20391 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:05.900524 20392 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.914814 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:05.910482 20420 consensus_queue.cc:1048] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:05.908959 20318 tablet_bootstrap.cc:492] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:05.919780 20391 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:05.905987 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:05.882251 20408 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:05.913164 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:05.939306 20048 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:05.939175 20424 consensus_queue.cc:1059] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:18:05.944453 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:05.948648 20198 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:05.949271 20322 tablet_bootstrap.cc:654] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:05.952492 20238 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:05.960114 20392 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:05.961216 20392 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:05.961421 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:05.961722 20392 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:05.965919 20239 leader_election.cc:343] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:05.963603 20016 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:05.965937 20320 tablet_bootstrap.cc:492] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:05.971536 20320 ts_tablet_manager.cc:1403] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.300s user 0.022s sys 0.015s
I20260317 16:18:05.971765 20408 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:05.966666 20392 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:05.974546 20320 raft_consensus.cc:359] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.975339 20320 raft_consensus.cc:385] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:18:05.951222 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 80789.5us
I20260317 16:18:05.979594 20419 raft_consensus.cc:3055] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:05.980029 20419 raft_consensus.cc:740] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:05.975629 20320 raft_consensus.cc:740] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:05.980854 20419 consensus_queue.cc:260] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:05.991829 20419 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:05.991775 20320 consensus_queue.cc:260] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:05.985448 20391 consensus_queue.cc:1048] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:05.986940 19947 catalog_manager.cc:5671] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:05.996016 20320 ts_tablet_manager.cc:1434] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.024s user 0.008s sys 0.001s
I20260317 16:18:06.011858 20408 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:06.012084 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 60363.1us
I20260317 16:18:06.012521 20408 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.014070 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.015041 20320 tablet_bootstrap.cc:492] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:06.017721 20408 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:06.019979 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.031127 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.033983 20198 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:06.036130 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:06.002652 19945 catalog_manager.cc:5671] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:06.034953 20392 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.043953 20408 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:06.059275 20322 tablet_bootstrap.cc:492] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:06.060223 20322 ts_tablet_manager.cc:1403] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.229s user 0.023s sys 0.015s
I20260317 16:18:06.060454 20320 tablet_bootstrap.cc:654] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.066439 20238 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:06.068624 20391 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:06.069069 20391 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:06.069442 20391 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.071399 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:06.072120 20198 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.073462 20322 raft_consensus.cc:359] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.074290 20322 raft_consensus.cc:385] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.075434 20322 raft_consensus.cc:740] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.076134 20322 consensus_queue.cc:260] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.078511 20322 ts_tablet_manager.cc:1434] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260317 16:18:06.079567 20322 tablet_bootstrap.cc:492] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:06.087236 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:06.088156 20049 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.089140 20322 tablet_bootstrap.cc:654] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.091897 20392 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:06.087806 20391 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.098510 20321 tablet_bootstrap.cc:492] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.106570 20318 tablet_bootstrap.cc:654] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.110813 20321 ts_tablet_manager.cc:1403] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.274s user 0.040s sys 0.045s
I20260317 16:18:06.118428 20321 raft_consensus.cc:359] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.125036 20321 raft_consensus.cc:385] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.125497 20321 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.127246 20321 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.134251 20391 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:06.136982 20318 tablet_bootstrap.cc:492] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:06.137527 20318 ts_tablet_manager.cc:1403] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.256s user 0.007s sys 0.028s
I20260317 16:18:06.140403 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:06.144250 20049 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:06.143136 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:06.147572 20320 tablet_bootstrap.cc:492] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:06.122380 20198 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:06.141587 20199 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.150614 20238 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:06.151856 20428 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:06.152338 20428 raft_consensus.cc:697] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:06.153175 20428 consensus_queue.cc:237] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.155216 20320 ts_tablet_manager.cc:1403] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.141s user 0.027s sys 0.004s
I20260317 16:18:06.161346 20318 raft_consensus.cc:359] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.162212 20318 raft_consensus.cc:385] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.159462 20320 raft_consensus.cc:359] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:06.163291 20239 leader_election.cc:343] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.163265 20320 raft_consensus.cc:385] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.166986 20318 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.167268 20320 raft_consensus.cc:740] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.168081 20318 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.168543 20320 consensus_queue.cc:260] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.170498 20322 tablet_bootstrap.cc:492] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:06.171207 20322 ts_tablet_manager.cc:1403] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.092s user 0.022s sys 0.022s
I20260317 16:18:06.174594 20322 raft_consensus.cc:359] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.176105 20322 raft_consensus.cc:385] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.170624 20318 ts_tablet_manager.cc:1434] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.030s user 0.008s sys 0.001s
I20260317 16:18:06.177054 20322 raft_consensus.cc:740] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.177940 20322 consensus_queue.cc:260] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.181290 20318 tablet_bootstrap.cc:492] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:06.183035 20320 ts_tablet_manager.cc:1434] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.027s user 0.013s sys 0.000s
I20260317 16:18:06.186165 20321 ts_tablet_manager.cc:1434] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.075s user 0.011s sys 0.000s
I20260317 16:18:06.192118 20199 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:06.194087 20238 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:06.193442 20320 tablet_bootstrap.cc:492] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:06.198956 20318 tablet_bootstrap.cc:654] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.206053 20320 tablet_bootstrap.cc:654] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.197753 20321 tablet_bootstrap.cc:492] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:06.208287 20428 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:06.210461 20428 raft_consensus.cc:697] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:06.213251 20408 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.209540 20322 ts_tablet_manager.cc:1434] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.038s user 0.008s sys 0.001s
I20260317 16:18:06.213879 20408 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.215270 20322 tablet_bootstrap.cc:492] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:06.216188 20408 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:06.213147 20428 consensus_queue.cc:237] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.217907 20321 tablet_bootstrap.cc:654] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.218168 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.218334 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.219738 20123 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:06.218981 20272 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:06.222880 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:06.223661 19946 catalog_manager.cc:5671] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:06.225209 20408 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:06.225885 20408 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:06.226426 20408 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.233305 20321 tablet_bootstrap.cc:492] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.233877 20321 ts_tablet_manager.cc:1403] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.001s
I20260317 16:18:06.236430 20321 raft_consensus.cc:359] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.237108 20321 raft_consensus.cc:385] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.237389 20321 raft_consensus.cc:740] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.238153 20321 consensus_queue.cc:260] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.243852 20322 tablet_bootstrap.cc:654] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.244004 20318 tablet_bootstrap.cc:492] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:06.244578 20408 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.245364 20318 ts_tablet_manager.cc:1403] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.066s user 0.030s sys 0.019s
I20260317 16:18:06.244138 20320 tablet_bootstrap.cc:492] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:06.246241 20320 ts_tablet_manager.cc:1403] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.007s
I20260317 16:18:06.248227 20318 raft_consensus.cc:359] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.248939 20318 raft_consensus.cc:385] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.249243 20318 raft_consensus.cc:740] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.249042 20320 raft_consensus.cc:359] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.250948 20320 raft_consensus.cc:385] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.250308 20318 consensus_queue.cc:260] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.252969 20318 ts_tablet_manager.cc:1434] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260317 16:18:06.253652 20408 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:06.254092 20318 tablet_bootstrap.cc:492] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:06.255142 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:06.251385 20320 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.255800 20123 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.259526 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:06.256471 20320 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.262183 20272 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.264393 20320 ts_tablet_manager.cc:1434] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260317 16:18:06.260607 20321 ts_tablet_manager.cc:1434] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.026s user 0.004s sys 0.004s
I20260317 16:18:06.266371 20321 tablet_bootstrap.cc:492] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:06.267637 20318 tablet_bootstrap.cc:654] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.267704 20320 tablet_bootstrap.cc:492] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:06.283833 20428 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.284940 20428 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.289477 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:06.291105 20238 leader_election.cc:343] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.295419 20321 tablet_bootstrap.cc:654] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.291642 20419 consensus_queue.cc:1048] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260317 16:18:06.301497 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:06.298177 19948 catalog_manager.cc:5671] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:06.309351 20320 tablet_bootstrap.cc:654] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:06.313485 20236 leader_election.cc:343] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.314059 20236 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:06.315050 20391 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:06.315659 20391 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.315984 20123 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:06.316087 20391 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.319000 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:06.319970 20408 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:06.320444 20408 raft_consensus.cc:697] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:06.299121 20428 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:06.327572 20391 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:06.335291 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:06.336371 20318 tablet_bootstrap.cc:492] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:06.331149 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.330964 20408 consensus_queue.cc:237] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:06.336694 20236 leader_election.cc:343] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:06.344981 20239 leader_election.cc:343] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.348373 20239 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:06.331390 20322 tablet_bootstrap.cc:492] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:06.350138 20428 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:06.350940 20322 ts_tablet_manager.cc:1403] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.136s user 0.030s sys 0.021s
I20260317 16:18:06.338277 20318 ts_tablet_manager.cc:1403] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.084s user 0.018s sys 0.014s
I20260317 16:18:06.369769 20049 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.372293 20402 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.372848 20402 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.374998 20402 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:06.377348 20402 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.378058 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.378547 20318 raft_consensus.cc:359] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.378855 20273 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:06.378135 20402 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.380018 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.380527 20091 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:06.381736 20199 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:06.386083 20402 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:06.387483 20318 raft_consensus.cc:385] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.388365 20318 raft_consensus.cc:740] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.389631 20318 consensus_queue.cc:260] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.391042 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.392448 20199 raft_consensus.cc:2393] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:06.391078 20402 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:06.394007 20402 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:06.394419 20402 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.395327 20318 ts_tablet_manager.cc:1434] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260317 16:18:06.400925 20318 tablet_bootstrap.cc:492] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:06.346511 20272 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:06.421463 20408 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.422124 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.422413 20408 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.426793 20321 tablet_bootstrap.cc:492] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.427345 20321 ts_tablet_manager.cc:1403] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.161s user 0.014s sys 0.005s
I20260317 16:18:06.428519 20091 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:06.361337 20322 raft_consensus.cc:359] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.430388 20447 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.431025 20322 raft_consensus.cc:385] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.430277 20321 raft_consensus.cc:359] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.431362 20322 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.431493 20321 raft_consensus.cc:385] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.432047 20321 raft_consensus.cc:740] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.433810 20424 consensus_queue.cc:1048] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:06.433746 20321 consensus_queue.cc:260] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.444833 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.443837 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.442283 20408 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:06.432113 20322 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.448179 20449 consensus_queue.cc:1048] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:06.451745 20016 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:06.454324 20199 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.474238 20321 ts_tablet_manager.cc:1434] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.047s user 0.009s sys 0.006s
I20260317 16:18:06.475492 20321 tablet_bootstrap.cc:492] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:06.438017 20402 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.480592 20408 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:06.481814 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:06.482648 20198 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.485859 20321 tablet_bootstrap.cc:654] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.488349 20320 tablet_bootstrap.cc:492] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:06.489136 20320 ts_tablet_manager.cc:1403] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.222s user 0.064s sys 0.041s
I20260317 16:18:06.489478 20318 tablet_bootstrap.cc:654] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.492031 20320 raft_consensus.cc:359] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.492864 20320 raft_consensus.cc:385] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.493167 20320 raft_consensus.cc:740] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.494174 20320 consensus_queue.cc:260] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.498973 20320 ts_tablet_manager.cc:1434] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260317 16:18:06.499594 20322 ts_tablet_manager.cc:1434] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.141s user 0.005s sys 0.006s
I20260317 16:18:06.502271 19948 catalog_manager.cc:5671] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:06.505257 20322 tablet_bootstrap.cc:492] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:06.506104 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:06.507179 20273 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.507810 20402 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:06.509990 20198 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:06.513761 20322 tablet_bootstrap.cc:654] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.515303 20090 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:06.516932 20402 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:06.517944 20402 raft_consensus.cc:697] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:06.518975 20402 consensus_queue.cc:237] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.524169 20321 tablet_bootstrap.cc:492] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.524945 20321 ts_tablet_manager.cc:1403] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.015s
I20260317 16:18:06.524860 20273 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:06.527472 20448 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.442379 20447 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:06.528223 20448 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.528671 20321 raft_consensus.cc:359] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.529426 20321 raft_consensus.cc:385] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.529729 20321 raft_consensus.cc:740] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.530409 20321 consensus_queue.cc:260] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.532707 20321 ts_tablet_manager.cc:1434] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 16:18:06.533639 20321 tablet_bootstrap.cc:492] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:06.538079 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:06.539215 20048 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:06.539551 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.540383 20198 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:06.540683 20088 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:06.539618 20448 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:06.541988 20447 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:06.542508 20447 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:06.543525 20408 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.543989 20321 tablet_bootstrap.cc:654] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.544085 20408 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.543941 20447 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.546698 20408 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:06.548184 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.502295 20320 tablet_bootstrap.cc:492] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:06.548959 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.548964 20272 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:06.551884 20016 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:06.553813 20016 leader_election.cc:343] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.554963 20408 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:06.555413 20408 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:06.555796 20408 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.559463 20322 tablet_bootstrap.cc:492] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:06.560578 20322 ts_tablet_manager.cc:1403] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.005s
I20260317 16:18:06.567556 20320 tablet_bootstrap.cc:654] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.572067 20322 raft_consensus.cc:359] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.572865 20322 raft_consensus.cc:385] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.573145 20322 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.573817 20322 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.578732 20408 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.593690 20318 tablet_bootstrap.cc:492] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:06.592900 20408 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:06.594238 20318 ts_tablet_manager.cc:1403] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.194s user 0.018s sys 0.000s
I20260317 16:18:06.594702 20428 consensus_queue.cc:1048] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:06.595163 20321 tablet_bootstrap.cc:492] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.595697 20321 ts_tablet_manager.cc:1403] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.062s user 0.025s sys 0.028s
I20260317 16:18:06.596269 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:06.596892 20273 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.598892 20321 raft_consensus.cc:359] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.599612 20321 raft_consensus.cc:385] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.599905 20321 raft_consensus.cc:740] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.600544 20321 consensus_queue.cc:260] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.600725 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:06.602442 20456 consensus_queue.cc:1048] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260317 16:18:06.603233 20321 ts_tablet_manager.cc:1434] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260317 16:18:06.603426 20016 leader_election.cc:343] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.605270 20321 tablet_bootstrap.cc:492] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:06.608356 20320 tablet_bootstrap.cc:492] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:06.609213 20320 ts_tablet_manager.cc:1403] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.001s
I20260317 16:18:06.611794 20318 raft_consensus.cc:359] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.612928 20321 tablet_bootstrap.cc:654] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.613162 20318 raft_consensus.cc:385] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.613781 20318 raft_consensus.cc:740] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.617731 20322 ts_tablet_manager.cc:1434] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.057s user 0.008s sys 0.002s
I20260317 16:18:06.623750 19948 catalog_manager.cc:5671] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:06.625840 20320 raft_consensus.cc:359] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.627528 20320 raft_consensus.cc:385] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.615921 20318 consensus_queue.cc:260] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.632342 20318 ts_tablet_manager.cc:1434] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.038s user 0.008s sys 0.001s
I20260317 16:18:06.633939 20428 consensus_queue.cc:1048] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:06.636196 20447 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.628036 20320 raft_consensus.cc:740] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.646279 20320 consensus_queue.cc:260] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.660995 20320 ts_tablet_manager.cc:1434] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.051s user 0.015s sys 0.000s
I20260317 16:18:06.668085 20322 tablet_bootstrap.cc:492] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:06.674350 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:06.675045 20048 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.679404 20318 tablet_bootstrap.cc:492] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:06.687450 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:06.688210 20197 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.696873 20391 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.697499 20391 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.700804 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.701517 20322 tablet_bootstrap.cc:654] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.709581 20320 tablet_bootstrap.cc:492] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
W20260317 16:18:06.718077 20239 leader_election.cc:343] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.674268 20273 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:06.715813 20408 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.719094 20408 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.719693 20420 consensus_queue.cc:1048] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:06.721259 20408 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:06.722189 20321 tablet_bootstrap.cc:492] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.721881 20428 consensus_queue.cc:1048] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:06.723863 20016 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:06.724395 20321 ts_tablet_manager.cc:1403] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.119s user 0.019s sys 0.014s
I20260317 16:18:06.726614 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.727535 20273 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:06.725390 20416 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.728201 20416 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.730330 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:06.732352 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.733206 20197 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:06.703984 20391 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:06.694561 20447 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:06.734167 20015 leader_election.cc:343] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.723914 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:06.730479 20416 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:06.726431 20048 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:06.725554 20408 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
W20260317 16:18:06.737509 20236 leader_election.cc:343] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.737987 20408 raft_consensus.cc:697] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:06.738641 20236 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:06.740689 20391 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:06.736495 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:06.736304 20195 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:06.739073 20090 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:06.748205 20015 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:06.752173 20318 tablet_bootstrap.cc:654] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.756309 20321 raft_consensus.cc:359] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.757300 20321 raft_consensus.cc:385] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.758651 20321 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.760792 20321 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:06.754032 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 53018.5us
I20260317 16:18:06.766500 20416 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:06.773629 20416 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:06.763841 20447 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:06.774698 20321 ts_tablet_manager.cc:1434] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.048s user 0.010s sys 0.002s
I20260317 16:18:06.774096 20416 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.776525 20447 raft_consensus.cc:697] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:06.777885 20447 consensus_queue.cc:237] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.751188 20408 consensus_queue.cc:237] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.780596 20456 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.781291 20456 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.786252 20456 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:06.787086 20321 tablet_bootstrap.cc:492] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:06.777662 20016 leader_election.cc:343] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:06.793581 20016 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:06.794318 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:06.797529 20320 tablet_bootstrap.cc:654] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.810084 20273 raft_consensus.cc:2393] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:06.824059 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 46150.9us
I20260317 16:18:06.827454 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:06.828337 20320 tablet_bootstrap.cc:492] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:06.828850 20320 ts_tablet_manager.cc:1403] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.120s user 0.016s sys 0.005s
I20260317 16:18:06.829679 20470 consensus_queue.cc:1048] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:06.820473 20408 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:06.841343 20320 raft_consensus.cc:359] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.844815 20320 raft_consensus.cc:385] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.845144 20320 raft_consensus.cc:740] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.847622 20320 consensus_queue.cc:260] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.849494 20091 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:06.850844 20447 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.857584 20320 ts_tablet_manager.cc:1434] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20260317 16:18:06.858831 20320 tablet_bootstrap.cc:492] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:06.856923 20322 tablet_bootstrap.cc:492] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:06.865343 20322 ts_tablet_manager.cc:1403] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.197s user 0.028s sys 0.019s
I20260317 16:18:06.868037 20416 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.872530 20321 tablet_bootstrap.cc:654] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.875903 20416 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:06.874027 19948 catalog_manager.cc:5671] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:06.868103 20322 raft_consensus.cc:359] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.904192 20322 raft_consensus.cc:385] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.904657 20322 raft_consensus.cc:740] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.905380 20322 consensus_queue.cc:260] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.885591 20318 tablet_bootstrap.cc:492] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:06.906929 20318 ts_tablet_manager.cc:1403] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.228s user 0.017s sys 0.005s
I20260317 16:18:06.908880 20321 tablet_bootstrap.cc:492] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.909426 20321 ts_tablet_manager.cc:1403] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.002s
I20260317 16:18:06.885680 20447 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:06.889006 20320 tablet_bootstrap.cc:654] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.891635 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:06.910404 20198 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.912079 20464 consensus_queue.cc:1048] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:06.912393 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:06.912994 20272 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:06.923787 20272 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:06.909615 20318 raft_consensus.cc:359] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.927067 20321 raft_consensus.cc:359] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.927652 20318 raft_consensus.cc:385] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.935416 20321 raft_consensus.cc:385] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.935819 20318 raft_consensus.cc:740] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.936300 20321 raft_consensus.cc:740] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.937218 20321 consensus_queue.cc:260] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.940524 20321 ts_tablet_manager.cc:1434] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.031s user 0.013s sys 0.005s
I20260317 16:18:06.940929 20016 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:06.941792 20321 tablet_bootstrap.cc:492] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:06.942339 20470 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:06.943527 20470 raft_consensus.cc:697] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:06.944592 20470 consensus_queue.cc:237] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:06.937247 20318 consensus_queue.cc:260] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:06.953603 20318 ts_tablet_manager.cc:1434] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260317 16:18:06.954613 20318 tablet_bootstrap.cc:492] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:06.954712 20322 ts_tablet_manager.cc:1434] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.089s user 0.008s sys 0.000s
I20260317 16:18:06.955817 20322 tablet_bootstrap.cc:492] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:06.962426 20318 tablet_bootstrap.cc:654] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.963116 20321 tablet_bootstrap.cc:654] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.962193 20198 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:06.974965 20321 tablet_bootstrap.cc:492] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:06.975561 20321 ts_tablet_manager.cc:1403] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.003s
I20260317 16:18:06.975754 20322 tablet_bootstrap.cc:654] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:06.984190 20318 tablet_bootstrap.cc:492] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:06.984766 20318 ts_tablet_manager.cc:1403] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.003s
I20260317 16:18:06.979220 20321 raft_consensus.cc:359] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.987963 20321 raft_consensus.cc:385] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.988425 20321 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.989879 20321 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:06.994954 20320 tablet_bootstrap.cc:492] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:06.994683 20318 raft_consensus.cc:359] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:06.995534 20320 ts_tablet_manager.cc:1403] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.000s
I20260317 16:18:06.995599 20318 raft_consensus.cc:385] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:06.996021 20318 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:06.996089 20447 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:06.996593 20447 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.001762 20321 ts_tablet_manager.cc:1434] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.026s user 0.007s sys 0.003s
I20260317 16:18:07.001303 20318 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.003248 20321 tablet_bootstrap.cc:492] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:07.004395 20320 raft_consensus.cc:359] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.005198 20320 raft_consensus.cc:385] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.005633 20320 raft_consensus.cc:740] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.006632 20320 consensus_queue.cc:260] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.001904 19946 catalog_manager.cc:5671] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:07.023490 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:07.023917 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.024507 20049 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:07.026357 20320 ts_tablet_manager.cc:1434] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.030s user 0.010s sys 0.000s
I20260317 16:18:06.999694 20447 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.030400 20320 tablet_bootstrap.cc:492] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:07.016713 19946 catalog_manager.cc:5671] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } }
W20260317 16:18:07.025911 20090 leader_election.cc:343] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.034592 20318 ts_tablet_manager.cc:1434] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.049s user 0.006s sys 0.005s
I20260317 16:18:07.046543 20318 tablet_bootstrap.cc:492] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:07.052417 20322 tablet_bootstrap.cc:492] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:07.053102 20322 ts_tablet_manager.cc:1403] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.097s user 0.017s sys 0.000s
I20260317 16:18:07.038597 20478 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.054373 20088 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:07.054601 20478 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.055584 20447 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.056070 20447 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.056452 20447 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.059947 20449 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:18:07.061193 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:07.066919 20321 tablet_bootstrap.cc:654] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.067684 20320 tablet_bootstrap.cc:654] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.067616 20470 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.071467 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.071645 20470 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.073997 20470 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:07.066220 20478 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.079077 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.079815 20198 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:07.080188 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:07.080947 20123 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
W20260317 16:18:07.082163 20238 leader_election.cc:343] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.083598 20015 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:07.084442 20318 tablet_bootstrap.cc:654] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.085381 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:07.086087 20273 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:07.089744 20470 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.107013 20470 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.123339 20470 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.149236 20485 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.124866 20239 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:07.151414 20478 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.151895 20478 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.151489 20447 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.152441 20478 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.153792 20447 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.155791 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:07.158999 20090 leader_election.cc:343] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.163271 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:07.164155 20049 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.169998 20478 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.172077 20318 tablet_bootstrap.cc:492] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:07.172658 20318 ts_tablet_manager.cc:1403] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.131s user 0.025s sys 0.014s
I20260317 16:18:07.174993 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:07.175400 20318 raft_consensus.cc:359] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.175952 20125 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.176193 20318 raft_consensus.cc:385] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.176645 20318 raft_consensus.cc:740] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.180971 20320 tablet_bootstrap.cc:492] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:07.181906 20320 ts_tablet_manager.cc:1403] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.154s user 0.025s sys 0.010s
I20260317 16:18:07.184883 20320 raft_consensus.cc:359] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.185925 20320 raft_consensus.cc:385] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.186239 20320 raft_consensus.cc:740] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.187178 20320 consensus_queue.cc:260] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.188694 20318 consensus_queue.cc:260] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.194962 20320 ts_tablet_manager.cc:1434] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260317 16:18:07.191519 20318 ts_tablet_manager.cc:1434] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260317 16:18:07.176113 20322 raft_consensus.cc:359] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.200196 20322 raft_consensus.cc:385] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.201267 20322 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.202585 20318 tablet_bootstrap.cc:492] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:07.203343 20322 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.204319 20320 tablet_bootstrap.cc:492] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:07.210014 20470 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.200763 20478 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.205654 20049 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:07.207268 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:07.215242 20424 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.215905 20424 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.221438 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:07.222872 20123 raft_consensus.cc:2393] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:07.224148 20238 leader_election.cc:343] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.221438 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:07.226131 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:07.227596 20048 raft_consensus.cc:2393] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:07.218230 20424 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:07.190996 20125 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:07.231395 20162 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:07.227963 20272 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.234449 20239 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:07.238863 20478 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.240280 20478 raft_consensus.cc:697] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:07.239326 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:07.238929 20424 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.241835 20478 consensus_queue.cc:237] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.250903 20408 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.251543 20408 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.253455 20408 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:07.255059 20088 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:07.255611 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:07.256291 20485 consensus_queue.cc:1048] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:18:07.257375 20016 leader_election.cc:343] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.257640 20272 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:07.257936 20485 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.241884 20199 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.258690 20485 raft_consensus.cc:697] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:07.261505 20485 consensus_queue.cc:237] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.239176 20321 tablet_bootstrap.cc:492] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:07.264106 20321 ts_tablet_manager.cc:1403] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.261s user 0.017s sys 0.007s
I20260317 16:18:07.266237 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:07.268712 20273 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:07.270459 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:07.272234 20408 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.272642 20408 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.273005 20408 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.273883 20456 consensus_queue.cc:1048] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.267035 20321 raft_consensus.cc:359] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.281035 20321 raft_consensus.cc:385] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.281637 20321 raft_consensus.cc:740] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.287444 20321 consensus_queue.cc:260] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.290469 19947 catalog_manager.cc:5671] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:07.213063 20322 ts_tablet_manager.cc:1434] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.160s user 0.012s sys 0.000s
I20260317 16:18:07.296043 20322 tablet_bootstrap.cc:492] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:07.300637 20321 ts_tablet_manager.cc:1434] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.036s user 0.003s sys 0.005s
I20260317 16:18:07.301918 20321 tablet_bootstrap.cc:492] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:07.303128 20320 tablet_bootstrap.cc:654] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.303041 20464 consensus_queue.cc:1048] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.305122 20485 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.224212 20470 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:07.305661 20485 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.284304 20318 tablet_bootstrap.cc:654] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.311506 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.312237 20195 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:07.313521 20090 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:07.313190 20408 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.315680 20408 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:07.315419 20449 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.317009 20424 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:07.317590 20449 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.318126 20449 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.323052 20321 tablet_bootstrap.cc:654] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.335740 20199 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:07.338860 20015 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:07.339665 19947 catalog_manager.cc:5671] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:07.341308 20470 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.341763 20322 tablet_bootstrap.cc:654] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.342442 20470 raft_consensus.cc:697] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:07.345363 20318 tablet_bootstrap.cc:492] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:07.346642 20318 ts_tablet_manager.cc:1403] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.147s user 0.025s sys 0.006s
I20260317 16:18:07.350018 20318 raft_consensus.cc:359] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.350484 20485 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.350955 20318 raft_consensus.cc:385] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.350739 20470 consensus_queue.cc:237] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.363283 20408 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.362406 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:07.358063 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:07.369246 20273 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.365604 20408 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.372087 20408 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:07.367767 20321 tablet_bootstrap.cc:492] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:07.372948 20321 ts_tablet_manager.cc:1403] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.004s
I20260317 16:18:07.375712 20321 raft_consensus.cc:359] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.376398 20321 raft_consensus.cc:385] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.376698 20321 raft_consensus.cc:740] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.377343 20321 consensus_queue.cc:260] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.379457 20321 ts_tablet_manager.cc:1434] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260317 16:18:07.351595 20318 raft_consensus.cc:740] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.380419 20321 tablet_bootstrap.cc:492] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:07.382180 20273 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:07.383330 20318 consensus_queue.cc:260] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.383870 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:07.384805 20049 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:07.385043 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:07.388513 20318 ts_tablet_manager.cc:1434] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
W20260317 16:18:07.388756 20016 leader_election.cc:343] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:07.389430 20016 leader_election.cc:343] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.389616 20318 tablet_bootstrap.cc:492] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:07.390578 20408 consensus_queue.cc:1048] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.394407 20449 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.396914 20449 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.397320 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:07.398653 20464 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.399405 20464 raft_consensus.cc:697] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:07.400341 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:07.400455 20464 consensus_queue.cc:237] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.401078 20273 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:07.402542 20321 tablet_bootstrap.cc:654] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.400493 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:07.404029 20199 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.405830 20318 tablet_bootstrap.cc:654] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.406277 20320 tablet_bootstrap.cc:492] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:07.406980 20320 ts_tablet_manager.cc:1403] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.207s user 0.024s sys 0.002s
I20260317 16:18:07.410655 20320 raft_consensus.cc:359] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.411468 20320 raft_consensus.cc:385] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.411738 20320 raft_consensus.cc:740] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.413757 20322 tablet_bootstrap.cc:492] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:07.414306 20322 ts_tablet_manager.cc:1403] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.017s
I20260317 16:18:07.412506 20320 consensus_queue.cc:260] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.417294 20322 raft_consensus.cc:359] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.418401 20322 raft_consensus.cc:385] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.418807 20322 raft_consensus.cc:740] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.419854 20322 consensus_queue.cc:260] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.428100 20322 ts_tablet_manager.cc:1434] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20260317 16:18:07.429090 20322 tablet_bootstrap.cc:492] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:07.430584 20320 ts_tablet_manager.cc:1434] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260317 16:18:07.438939 20320 tablet_bootstrap.cc:492] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:07.441509 20016 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:07.441958 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:07.442749 20049 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.448194 20320 tablet_bootstrap.cc:654] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.451467 20470 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.453837 20470 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.454236 20470 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.454998 19947 catalog_manager.cc:5671] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:07.462980 20322 tablet_bootstrap.cc:654] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.463761 20470 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.466048 20470 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:07.468238 20049 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:07.457242 20199 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:07.470486 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:07.470747 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:07.471359 20272 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.476148 20090 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:07.495867 20449 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.496424 20449 raft_consensus.cc:697] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:07.504817 20447 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:07.508922 20447 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.514783 20464 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.515002 20449 consensus_queue.cc:237] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.515690 20464 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.521049 20464 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.523663 20447 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.523877 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.524657 20198 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:07.526508 20015 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:07.526276 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.527618 20195 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:07.528254 20470 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:07.529102 20470 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.528379 20272 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:07.529364 20090 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:07.529443 20470 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:07.534296 20016 leader_election.cc:343] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:07.535373 20447 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:07.535888 20447 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:07.536334 20447 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:07.537743 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:07.561097 20470 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.561679 20428 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.564728 20428 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.569379 20428 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.574150 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:07.575368 20124 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:07.576304 20470 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.577580 20239 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:07.578217 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:07.580504 20195 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:07.572197 20447 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.600615 20123 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:07.593413 20428 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:07.608595 20321 tablet_bootstrap.cc:492] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:07.609150 20321 ts_tablet_manager.cc:1403] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.229s user 0.024s sys 0.006s
I20260317 16:18:07.611613 20320 tablet_bootstrap.cc:492] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:07.603250 20428 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:07.612310 20320 ts_tablet_manager.cc:1403] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.174s user 0.038s sys 0.031s
I20260317 16:18:07.613641 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:07.604893 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:07.620536 20048 raft_consensus.cc:2368] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:07.614849 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:07.617129 20447 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.648931 20318 tablet_bootstrap.cc:492] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:07.649545 20318 ts_tablet_manager.cc:1403] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.260s user 0.047s sys 0.095s
I20260317 16:18:07.650364 20320 raft_consensus.cc:359] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.647665 20428 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.652284 20320 raft_consensus.cc:385] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.654814 20320 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.664779 20320 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.672633 20320 ts_tablet_manager.cc:1434] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.054s user 0.012s sys 0.001s
I20260317 16:18:07.674530 20320 tablet_bootstrap.cc:492] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:07.681969 20195 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:07.688364 20322 tablet_bootstrap.cc:492] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:07.688920 20322 ts_tablet_manager.cc:1403] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.260s user 0.020s sys 0.028s
I20260317 16:18:07.691026 20322 raft_consensus.cc:359] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.691843 20322 raft_consensus.cc:385] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.692066 20320 tablet_bootstrap.cc:654] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.692209 20322 raft_consensus.cc:740] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.693835 20322 consensus_queue.cc:260] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.694577 20428 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.698828 20428 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:07.701154 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:07.701536 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:07.701915 20124 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.707574 20322 ts_tablet_manager.cc:1434] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20260317 16:18:07.708709 20322 tablet_bootstrap.cc:492] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:07.709992 20198 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:07.712466 20124 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:07.713979 20239 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:07.714644 20322 tablet_bootstrap.cc:654] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.715003 20420 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.715488 20420 raft_consensus.cc:697] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:07.716416 20420 consensus_queue.cc:237] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.745739 20485 consensus_queue.cc:1048] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:07.650440 20199 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:07.766860 20318 raft_consensus.cc:359] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.768203 20318 raft_consensus.cc:385] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.768596 20318 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.769572 20318 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.772652 20318 ts_tablet_manager.cc:1434] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.122s user 0.006s sys 0.002s
I20260317 16:18:07.777827 20318 tablet_bootstrap.cc:492] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
W20260317 16:18:07.782406 20506 log.cc:927] Time spent T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260317 16:18:07.806437 20322 tablet_bootstrap.cc:492] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:07.806423 20320 tablet_bootstrap.cc:492] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:07.807103 20322 ts_tablet_manager.cc:1403] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.007s
I20260317 16:18:07.807446 20320 ts_tablet_manager.cc:1403] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.133s user 0.023s sys 0.031s
I20260317 16:18:07.810456 20322 raft_consensus.cc:359] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.811314 20322 raft_consensus.cc:385] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.811620 20322 raft_consensus.cc:740] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
W20260317 16:18:07.811820 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 158269us
I20260317 16:18:07.812604 20016 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:07.812469 20322 consensus_queue.cc:260] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.815219 20322 ts_tablet_manager.cc:1434] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260317 16:18:07.816730 20420 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.817512 20318 tablet_bootstrap.cc:654] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.818737 20321 raft_consensus.cc:359] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.819475 20321 raft_consensus.cc:385] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.820061 20321 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.820760 20321 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.817641 20470 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:07.829806 20464 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.830475 20464 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.833448 20464 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:07.838994 20478 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.840034 20478 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.840559 20470 raft_consensus.cc:697] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:07.842821 20470 consensus_queue.cc:237] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.843852 20478 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:07.816455 20322 tablet_bootstrap.cc:492] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:07.852147 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:07.853139 20047 raft_consensus.cc:2368] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:07.860400 20321 ts_tablet_manager.cc:1434] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.251s user 0.010s sys 0.000s
I20260317 16:18:07.823071 20320 raft_consensus.cc:359] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.862296 20320 raft_consensus.cc:385] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.862655 20320 raft_consensus.cc:740] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.866739 20464 consensus_queue.cc:1048] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.863407 20320 consensus_queue.cc:260] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:07.928406 20322 tablet_bootstrap.cc:654] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.928603 20320 ts_tablet_manager.cc:1434] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.120s user 0.012s sys 0.002s
I20260317 16:18:07.929888 20320 tablet_bootstrap.cc:492] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:07.934032 20318 tablet_bootstrap.cc:492] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:07.934728 20318 ts_tablet_manager.cc:1403] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.161s user 0.039s sys 0.040s
I20260317 16:18:07.938076 20320 tablet_bootstrap.cc:654] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.940271 20318 raft_consensus.cc:359] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.941236 20318 raft_consensus.cc:385] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.941668 20318 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.943091 20318 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.946238 20318 ts_tablet_manager.cc:1434] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260317 16:18:07.946632 20322 tablet_bootstrap.cc:492] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:07.947232 20322 ts_tablet_manager.cc:1403] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.131s user 0.019s sys 0.001s
I20260317 16:18:07.947395 20318 tablet_bootstrap.cc:492] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:07.950810 20322 raft_consensus.cc:359] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.951586 20322 raft_consensus.cc:385] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.947590 20320 tablet_bootstrap.cc:492] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:07.951874 20322 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.952505 20320 ts_tablet_manager.cc:1403] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.007s
I20260317 16:18:07.952841 20322 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.955668 20322 ts_tablet_manager.cc:1434] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260317 16:18:07.955721 20320 raft_consensus.cc:359] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.956665 20322 tablet_bootstrap.cc:492] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:07.956792 20320 raft_consensus.cc:385] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.957298 20320 raft_consensus.cc:740] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.958005 20320 consensus_queue.cc:260] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:07.959964 20320 ts_tablet_manager.cc:1434] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 16:18:07.960732 20088 leader_election.cc:400] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:07.960942 20320 tablet_bootstrap.cc:492] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:07.961844 20088 leader_election.cc:400] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:07.962174 20088 leader_election.cc:403] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260317 16:18:07.962821 20449 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:07.963871 20318 tablet_bootstrap.cc:654] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.964069 20495 consensus_queue.cc:1048] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:07.972684 20320 tablet_bootstrap.cc:654] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.974786 20495 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:07.975385 20495 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:07.977941 20495 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:07.979436 20449 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:07.980652 20318 tablet_bootstrap.cc:492] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:07.981168 20318 ts_tablet_manager.cc:1403] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.002s
I20260317 16:18:07.983541 20322 tablet_bootstrap.cc:654] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:07.984071 20318 raft_consensus.cc:359] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.984900 20318 raft_consensus.cc:385] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:07.985219 20318 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:07.985980 20318 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:07.986989 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:07.988233 20318 ts_tablet_manager.cc:1434] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260317 16:18:07.989251 20318 tablet_bootstrap.cc:492] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:07.997059 20320 tablet_bootstrap.cc:492] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:07.998216 20320 ts_tablet_manager.cc:1403] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.002s
I20260317 16:18:07.998880 20322 tablet_bootstrap.cc:492] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:07.999634 20322 ts_tablet_manager.cc:1403] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.003s
I20260317 16:18:08.003115 20320 raft_consensus.cc:359] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.004082 20320 raft_consensus.cc:385] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.004459 20320 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.005298 20320 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.007651 20318 tablet_bootstrap.cc:654] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.008219 20320 ts_tablet_manager.cc:1434] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260317 16:18:08.009534 20320 tablet_bootstrap.cc:492] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:08.013108 20322 raft_consensus.cc:359] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.015897 20320 tablet_bootstrap.cc:654] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.015540 20318 tablet_bootstrap.cc:492] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:08.016633 20318 ts_tablet_manager.cc:1403] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.001s
I20260317 16:18:08.019269 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.020023 20199 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:08.016433 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:08.028962 20048 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:08.030887 20236 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:08.031973 20420 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:08.032478 20420 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:08.032835 20420 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.034874 20519 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.035521 20519 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.038170 20519 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:08.015951 20322 raft_consensus.cc:385] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.073525 20322 raft_consensus.cc:740] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.076320 20322 consensus_queue.cc:260] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.025151 20318 raft_consensus.cc:359] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.079327 20318 raft_consensus.cc:385] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.079909 20318 raft_consensus.cc:740] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.080076 20322 ts_tablet_manager.cc:1434] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.072s user 0.006s sys 0.002s
I20260317 16:18:08.081532 20322 tablet_bootstrap.cc:492] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:08.082746 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:08.082819 20449 consensus_queue.cc:1048] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.083673 20048 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:08.084178 20485 consensus_queue.cc:1048] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:08.084738 20162 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:08.085754 20519 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:08.099511 20519 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:08.100064 20519 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.101976 20320 tablet_bootstrap.cc:492] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:08.102505 20320 ts_tablet_manager.cc:1403] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.093s user 0.022s sys 0.005s
I20260317 16:18:08.092424 20318 consensus_queue.cc:260] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.095394 20420 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.108629 20420 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:08.109510 20318 ts_tablet_manager.cc:1434] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.092s user 0.007s sys 0.000s
I20260317 16:18:08.110213 20320 raft_consensus.cc:359] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.110889 20318 tablet_bootstrap.cc:492] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:08.110960 20320 raft_consensus.cc:385] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.111343 20320 raft_consensus.cc:740] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.112298 20320 consensus_queue.cc:260] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.115489 20519 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.128407 20199 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.130394 20519 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:08.141063 20449 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.145289 20318 tablet_bootstrap.cc:654] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.142416 20449 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.155442 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:08.158733 20050 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.159420 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:08.167750 20322 tablet_bootstrap.cc:654] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.179884 20495 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.180564 20495 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:08.184115 20122 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.187315 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:08.188244 20273 raft_consensus.cc:2393] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:08.190402 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:08.191457 20049 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.192677 20449 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:08.170123 20194 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.198839 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.205087 20193 raft_consensus.cc:2393] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:08.228375 20015 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:08.229363 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:08.230113 20119 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.236219 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.241540 20526 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:08.241719 20447 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.242178 20526 raft_consensus.cc:697] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:08.242481 20447 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.251297 20447 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:08.246467 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:08.256343 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.256992 20196 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:08.256176 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:08.258504 20090 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:08.272320 20088 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:08.282016 20050 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:08.282109 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:08.283308 20273 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:08.283885 20236 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:08.285547 20534 consensus_queue.cc:1048] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.278586 20049 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:08.282347 20533 consensus_queue.cc:1048] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20260317 16:18:08.334987 20162 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:08.349501 20527 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:08.350087 20527 raft_consensus.cc:697] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:08.355540 20447 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:08.356088 20447 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:08.356448 20447 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.345214 20526 consensus_queue.cc:237] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.363168 20527 consensus_queue.cc:237] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:08.367813 20464 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:18:08.329244 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 44348us
I20260317 16:18:08.378826 20533 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:08.384054 20533 raft_consensus.cc:697] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:08.389676 20538 consensus_queue.cc:1048] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.386183 20533 consensus_queue.cc:237] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:08.336365 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 64173.4us
I20260317 16:18:08.380877 20119 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:08.477305 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:08.492157 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:08.494472 20535 raft_consensus.cc:2749] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:08.472903 20194 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:08.515017 20125 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:08.501201 20527 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.492636 20495 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:08.560119 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:08.561092 20272 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:08.568755 20322 tablet_bootstrap.cc:492] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:08.569314 20322 ts_tablet_manager.cc:1403] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.488s user 0.055s sys 0.033s
I20260317 16:18:08.536005 20120 raft_consensus.cc:2376] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1. Re-sending same reply.
I20260317 16:18:08.530234 20527 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.586933 20537 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.587558 20537 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.590131 20537 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:08.579125 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:08.590400 20322 raft_consensus.cc:359] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.591248 20322 raft_consensus.cc:385] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.592311 20322 raft_consensus.cc:740] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.594390 20322 consensus_queue.cc:260] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.598500 20322 ts_tablet_manager.cc:1434] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.029s user 0.007s sys 0.003s
I20260317 16:18:08.599080 20527 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:08.599841 20322 tablet_bootstrap.cc:492] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:08.603149 20125 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.605299 20091 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:08.608453 20322 tablet_bootstrap.cc:654] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.616884 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.617735 20194 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:08.595036 20016 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:08.624681 20526 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:08.624920 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 44285.4us
I20260317 16:18:08.639191 20470 consensus_queue.cc:1048] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.638520 20238 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:08.643505 20478 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:08.644100 20478 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:08.644469 20478 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:08.653717 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 48538.7us
I20260317 16:18:08.652887 20535 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:08.659940 20526 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.659893 20478 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.662958 20478 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:08.664009 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:08.664671 20194 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.578567 20533 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20260317 16:18:08.673933 20320 ts_tablet_manager.cc:1434] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.571s user 0.037s sys 0.014s
W20260317 16:18:08.674307 20320 ts_tablet_manager.cc:1467] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Tablet startup took 571ms
I20260317 16:18:08.675036 20194 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:08.676836 20238 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:08.677636 20478 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:08.678103 20478 raft_consensus.cc:697] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:08.678978 20478 consensus_queue.cc:237] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:08.683398 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:08.684237 19947 catalog_manager.cc:5671] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:08.687608 20321 tablet_bootstrap.cc:492] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:08.674621 20320 ts_tablet_manager.cc:1469] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Trace:
0317 16:18:03.545345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:03.549851 (+ 4506us) service_pool.cc:225] Handling call
0317 16:18:03.568788 (+ 18937us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:03.568900 (+ 112us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:03.639960 (+ 71060us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:03.652523 (+ 12563us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:03.653495 (+ 972us) inbound_call.cc:173] Queueing success response
0317 16:18:08.009000 (+4355505us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:08.102814 (+ 93814us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:08.103350 (+ 536us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:08.103922 (+ 572us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":18271,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"tablet-open.queue_time_us":4355611,"thread_start_us":61943,"threads_started":1}
I20260317 16:18:08.693811 20320 tablet_bootstrap.cc:492] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
W20260317 16:18:08.714838 20318 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.011s sys 0.017s
W20260317 16:18:08.715379 20318 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.567s user 0.011s sys 0.017s
I20260317 16:18:08.717154 20470 consensus_queue.cc:1048] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.718317 20318 tablet_bootstrap.cc:492] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:08.718897 20318 ts_tablet_manager.cc:1403] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.608s user 0.029s sys 0.019s
I20260317 16:18:08.720538 20320 tablet_bootstrap.cc:654] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:08.724437 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 59427.3us
I20260317 16:18:08.727145 20088 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:08.728456 20449 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:08.731146 20447 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.734069 20447 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:08.743127 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:08.744083 20268 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:08.745981 20321 tablet_bootstrap.cc:654] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.748809 20318 raft_consensus.cc:359] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:08.749975 20318 raft_consensus.cc:385] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.750320 20318 raft_consensus.cc:740] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.751322 20318 consensus_queue.cc:260] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:08.760852 20322 tablet_bootstrap.cc:492] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:08.761683 20322 ts_tablet_manager.cc:1403] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.162s user 0.025s sys 0.033s
I20260317 16:18:08.766450 20322 raft_consensus.cc:359] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:08.769629 20322 raft_consensus.cc:385] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.773274 20322 raft_consensus.cc:740] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.784153 20318 ts_tablet_manager.cc:1434] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.065s user 0.006s sys 0.003s
I20260317 16:18:08.786098 20318 tablet_bootstrap.cc:492] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:08.785835 20322 consensus_queue.cc:260] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:08.804307 20322 ts_tablet_manager.cc:1434] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.042s user 0.009s sys 0.001s
I20260317 16:18:08.804843 20547 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.805395 20547 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.807534 20547 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:08.805347 20322 tablet_bootstrap.cc:492] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:08.817576 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:08.819897 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:08.820851 20318 tablet_bootstrap.cc:654] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.839107 20320 tablet_bootstrap.cc:492] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:08.839615 20320 ts_tablet_manager.cc:1403] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.146s user 0.033s sys 0.007s
I20260317 16:18:08.860880 20322 tablet_bootstrap.cc:654] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:08.845017 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:08.872339 20478 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.874838 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:08.881068 20478 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:08.887773 20268 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:08.900617 20321 tablet_bootstrap.cc:492] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:08.901293 20321 ts_tablet_manager.cc:1403] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.214s user 0.020s sys 0.014s
I20260317 16:18:08.906221 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:08.907429 20196 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:08.911247 20162 leader_election.cc:343] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:08.913192 20537 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.916854 20321 raft_consensus.cc:359] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:08.917490 20236 leader_election.cc:343] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:08.917730 20321 raft_consensus.cc:385] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.918319 20321 raft_consensus.cc:740] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.917774 20537 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:08.923041 20236 leader_election.cc:343] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:08.924705 20537 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:08.904994 20416 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260317 16:18:08.930377 20534 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:08.932500 20091 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:08.935163 20321 consensus_queue.cc:260] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:08.937328 20428 consensus_queue.cc:1048] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 16:18:08.943018 20478 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:08.945084 20320 raft_consensus.cc:359] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.947162 20320 raft_consensus.cc:385] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:08.947670 20320 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:08.951272 20320 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.950668 20449 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:08.964574 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.967340 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:08.988406 20420 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:08.989137 20420 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:08.994891 20320 ts_tablet_manager.cc:1434] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.155s user 0.014s sys 0.000s
I20260317 16:18:08.996994 20320 tablet_bootstrap.cc:492] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:09.006188 20424 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 16:18:09.014674 20321 ts_tablet_manager.cc:1434] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.113s user 0.012s sys 0.000s
I20260317 16:18:09.016013 20321 tablet_bootstrap.cc:492] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:08.991734 20449 raft_consensus.cc:697] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:09.022558 20196 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:09.024058 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:08.991142 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 53304.8us
I20260317 16:18:09.034298 20322 tablet_bootstrap.cc:492] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:09.035009 20322 ts_tablet_manager.cc:1403] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.230s user 0.017s sys 0.032s
W20260317 16:18:09.009104 20238 leader_election.cc:343] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.043056 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:09.043929 20124 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:09.043591 20449 consensus_queue.cc:237] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:09.051800 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 47372.5us
I20260317 16:18:09.070389 20420 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:09.072635 20562 log.cc:927] Time spent T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260317 16:18:09.074131 20318 tablet_bootstrap.cc:492] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:09.074608 20321 tablet_bootstrap.cc:654] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.075315 20124 raft_consensus.cc:1275] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:09.076606 20318 ts_tablet_manager.cc:1403] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.291s user 0.015s sys 0.012s
I20260317 16:18:09.068276 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:09.078205 20196 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:09.029728 20090 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:09.082672 20447 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:09.082912 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 68076.1us
I20260317 16:18:09.083429 20318 raft_consensus.cc:359] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.084448 20318 raft_consensus.cc:385] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.084836 20318 raft_consensus.cc:740] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.085851 20318 consensus_queue.cc:260] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:09.088130 20239 leader_election.cc:343] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.092300 20320 tablet_bootstrap.cc:654] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.094980 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:09.110440 20318 ts_tablet_manager.cc:1434] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.031s user 0.006s sys 0.002s
I20260317 16:18:09.111809 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:09.112125 20318 tablet_bootstrap.cc:492] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:09.112738 20050 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:09.113605 20239 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:09.114593 20537 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:09.115159 20239 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:09.116340 20420 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:09.116739 20420 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:09.117085 20420 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:09.117710 20534 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:09.118248 20534 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.119562 20322 raft_consensus.cc:359] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.116955 20238 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:09.125403 20428 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:09.120399 20534 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:09.125844 20428 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:09.126230 20428 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:09.127998 20424 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:18:09.130494 20236 leader_election.cc:343] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.121352 20318 tablet_bootstrap.cc:654] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.089394 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:09.123766 20322 raft_consensus.cc:385] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.134449 20322 raft_consensus.cc:740] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.134747 20165 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:09.136264 20424 raft_consensus.cc:2749] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:09.137372 20322 consensus_queue.cc:260] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.148166 20534 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260317 16:18:09.157951 20537 consensus_queue.cc:1048] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 16:18:09.141558 20478 consensus_queue.cc:1048] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260317 16:18:09.160859 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:09.161016 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:09.162566 20125 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:09.177886 20322 ts_tablet_manager.cc:1434] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.142s user 0.009s sys 0.001s
I20260317 16:18:09.181964 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:09.184720 20322 tablet_bootstrap.cc:492] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:09.195261 20534 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:09.196353 20534 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:09.196834 20534 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:09.218977 20534 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.221745 20534 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:09.223668 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:09.224299 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:09.224987 20124 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:09.231820 20321 tablet_bootstrap.cc:492] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:09.232551 20321 ts_tablet_manager.cc:1403] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.217s user 0.021s sys 0.017s
I20260317 16:18:09.236500 19945 catalog_manager.cc:5671] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
W20260317 16:18:09.238065 20294 debug-util.cc:398] Leaking SignalData structure 0x7b0800349900 after lost signal to thread 16882
W20260317 16:18:09.239300 20294 debug-util.cc:398] Leaking SignalData structure 0x7b080035b3c0 after lost signal to thread 19982
W20260317 16:18:09.240562 20294 debug-util.cc:398] Leaking SignalData structure 0x7b080030a660 after lost signal to thread 20074
W20260317 16:18:09.241683 20294 debug-util.cc:398] Leaking SignalData structure 0x7b080018e1a0 after lost signal to thread 20149
I20260317 16:18:09.241143 19948 catalog_manager.cc:5671] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } }
W20260317 16:18:09.243330 20294 debug-util.cc:398] Leaking SignalData structure 0x7b080015dd40 after lost signal to thread 20223
I20260317 16:18:09.247723 19948 catalog_manager.cc:5671] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:09.259744 19948 catalog_manager.cc:5671] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:09.250296 20537 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:09.268253 20194 raft_consensus.cc:1275] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:09.268000 20537 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.229745 20420 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.305719 20318 tablet_bootstrap.cc:492] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:09.307222 20318 ts_tablet_manager.cc:1403] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.195s user 0.013s sys 0.013s
W20260317 16:18:09.289896 20236 consensus_peers.cc:597] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 -> Peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Couldn't send request to peer 9a3afb098d6a45f0b405566771a1157b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 16:18:09.317633 20124 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:09.318419 20428 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:09.323217 20585 log.cc:927] Time spent T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.105s user 0.000s sys 0.004s
I20260317 16:18:09.337097 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:09.305562 20416 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260317 16:18:09.337831 20270 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:09.328616 20537 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:09.352625 20416 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:09.353578 20416 raft_consensus.cc:697] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:09.355770 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:09.354364 20582 log.cc:927] Time spent T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.138s user 0.001s sys 0.001s
I20260317 16:18:09.372648 20420 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:09.363234 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:09.379467 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:09.379591 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:09.383002 20238 leader_election.cc:343] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.383494 20122 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:09.354517 20416 consensus_queue.cc:237] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.401278 20320 tablet_bootstrap.cc:492] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:09.402125 20320 ts_tablet_manager.cc:1403] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.406s user 0.043s sys 0.036s
I20260317 16:18:09.348752 20428 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:09.403024 20321 raft_consensus.cc:359] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.412387 20321 raft_consensus.cc:385] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.412966 20321 raft_consensus.cc:740] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.414378 20321 consensus_queue.cc:260] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.439201 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:09.439770 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:09.440356 20197 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:09.441520 20236 leader_election.cc:343] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.478092 20321 ts_tablet_manager.cc:1434] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.245s user 0.013s sys 0.001s
I20260317 16:18:09.479331 20236 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:09.488471 20321 tablet_bootstrap.cc:492] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:09.490847 20322 tablet_bootstrap.cc:654] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.491485 20318 raft_consensus.cc:359] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.494009 20318 raft_consensus.cc:385] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.490113 20533 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:09.497193 20318 raft_consensus.cc:740] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.511488 20318 consensus_queue.cc:260] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.511931 20320 raft_consensus.cc:359] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.512970 20320 raft_consensus.cc:385] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.513516 20320 raft_consensus.cc:740] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
W20260317 16:18:09.517956 20589 log.cc:927] Time spent T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
W20260317 16:18:09.516616 20239 leader_election.cc:343] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.514269 20320 consensus_queue.cc:260] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.520841 20239 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:09.522116 20537 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:09.535909 20320 ts_tablet_manager.cc:1434] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.133s user 0.008s sys 0.002s
I20260317 16:18:09.537083 20320 tablet_bootstrap.cc:492] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:09.544318 19946 catalog_manager.cc:5671] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:09.554206 20318 ts_tablet_manager.cc:1434] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.241s user 0.014s sys 0.002s
I20260317 16:18:09.561882 20318 tablet_bootstrap.cc:492] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:09.552150 20545 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:09.572271 20545 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.575712 20545 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:09.579918 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:09.587795 20088 leader_election.cc:343] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.594969 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:09.605823 20322 tablet_bootstrap.cc:492] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:09.606469 20322 ts_tablet_manager.cc:1403] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.426s user 0.021s sys 0.005s
I20260317 16:18:09.640131 20318 tablet_bootstrap.cc:654] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:09.612375 20091 leader_election.cc:343] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.640945 20091 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:09.645754 20545 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:09.647135 20320 tablet_bootstrap.cc:654] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.647753 20545 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:09.648290 20545 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.655225 20322 raft_consensus.cc:359] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.653913 20197 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:09.667188 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:09.670949 20238 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:09.672457 20420 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:09.688705 20321 tablet_bootstrap.cc:654] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.689265 19947 catalog_manager.cc:5671] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } }
W20260317 16:18:09.692416 20090 leader_election.cc:343] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.673007 20420 raft_consensus.cc:697] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:09.656272 20322 raft_consensus.cc:385] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.704151 20322 raft_consensus.cc:740] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.704257 20420 consensus_queue.cc:237] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:09.702569 20545 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:09.705240 20322 consensus_queue.cc:260] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.702232 19947 catalog_manager.cc:5671] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:09.711727 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:09.712497 20268 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:09.714896 20091 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:09.717797 20322 ts_tablet_manager.cc:1434] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.110s user 0.013s sys 0.000s
I20260317 16:18:09.719089 20322 tablet_bootstrap.cc:492] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:09.710345 19947 catalog_manager.cc:5671] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 1 to 3, leader changed from 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195) to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 3 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } }
I20260317 16:18:09.730695 20545 raft_consensus.cc:2804] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:09.731266 20545 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:09.731657 20545 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:09.715570 20447 consensus_queue.cc:1048] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260317 16:18:09.758191 20449 consensus_queue.cc:1048] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:09.810161 20320 tablet_bootstrap.cc:492] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:09.810637 20320 ts_tablet_manager.cc:1403] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.274s user 0.063s sys 0.080s
I20260317 16:18:09.813649 20320 raft_consensus.cc:359] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.814138 20320 raft_consensus.cc:385] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.814448 20320 raft_consensus.cc:740] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.815151 20320 consensus_queue.cc:260] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.829072 20320 ts_tablet_manager.cc:1434] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260317 16:18:09.830472 20320 tablet_bootstrap.cc:492] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:09.833972 20318 tablet_bootstrap.cc:492] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:09.834430 20318 ts_tablet_manager.cc:1403] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.275s user 0.029s sys 0.012s
I20260317 16:18:09.841428 20424 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:09.842063 20424 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.845669 20321 tablet_bootstrap.cc:492] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:09.852368 20596 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:09.852993 20596 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.854256 20321 ts_tablet_manager.cc:1403] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.374s user 0.014s sys 0.023s
I20260317 16:18:09.863427 20596 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:09.867475 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:09.870193 20165 leader_election.cc:343] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:09.870361 20545 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:09.869417 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:09.872110 20424 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:09.873795 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:09.875622 20270 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:09.877624 20165 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:09.879739 20318 raft_consensus.cc:359] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.880527 20318 raft_consensus.cc:385] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.880945 20424 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:09.881423 20424 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:09.882151 20424 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:09.864053 20321 raft_consensus.cc:359] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.887184 20321 raft_consensus.cc:385] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:09.881322 20318 raft_consensus.cc:740] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.888093 20321 raft_consensus.cc:740] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:09.891729 20321 consensus_queue.cc:260] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:09.892038 20318 consensus_queue.cc:260] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:09.916383 20318 ts_tablet_manager.cc:1434] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.082s user 0.009s sys 0.001s
I20260317 16:18:09.907840 20321 ts_tablet_manager.cc:1434] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.052s user 0.010s sys 0.000s
I20260317 16:18:10.066023 20318 tablet_bootstrap.cc:492] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:09.912933 20320 tablet_bootstrap.cc:654] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:10.068928 20321 tablet_bootstrap.cc:492] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:09.923635 20322 tablet_bootstrap.cc:654] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:09.911476 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:09.925411 20545 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:10.139326 20318 tablet_bootstrap.cc:654] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:10.138747 20120 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:10.121647 20538 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:10.141577 20538 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:10.177146 20424 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:10.147423 20538 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:10.205556 20424 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:10.216364 20318 tablet_bootstrap.cc:492] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:10.217329 20318 ts_tablet_manager.cc:1403] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.152s user 0.035s sys 0.035s
I20260317 16:18:10.229351 20318 raft_consensus.cc:359] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:10.231384 20318 raft_consensus.cc:385] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:10.232004 20318 raft_consensus.cc:740] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:10.234637 20321 tablet_bootstrap.cc:654] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:10.235503 20318 consensus_queue.cc:260] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:10.238559 20318 ts_tablet_manager.cc:1434] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.021s user 0.012s sys 0.000s
I20260317 16:18:10.239743 20318 tablet_bootstrap.cc:492] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:11.197280 20609 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.197960 20609 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.200901 20609 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:11.244894 20220 debug-util.cc:398] Leaking SignalData structure 0x7b080025c460 after lost signal to thread 16882
W20260317 16:18:09.899158 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 54744.6us
I20260317 16:18:11.273191 20611 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.273825 20611 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.298309 20611 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.300117 20611 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:11.300688 20611 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:11.732244 20013 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.000s sys 0.008s
W20260317 16:18:11.732843 20013 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s user 0.001s sys 0.008s
I20260317 16:18:11.745031 20538 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.745954 20538 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.756608 20464 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:11.770787 20538 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.771349 20538 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:11.771272 20464 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.772094 20538 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.774174 20538 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.760373 20614 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.767446 20608 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:11.774986 20608 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.775424 20611 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.776476 20322 tablet_bootstrap.cc:492] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:11.777000 20608 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.776906 20464 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:11.769134 20089 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.001s sys 0.007s
W20260317 16:18:11.778049 20089 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.001s sys 0.007s
I20260317 16:18:11.779027 20611 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.780107 20613 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:11.779879 20611 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.780920 20613 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.783895 20613 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.784183 20610 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.784837 20612 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.784929 20610 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.785662 20612 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.788111 20612 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.788691 20610 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.783854 20611 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.774665 20538 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.790596 20538 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.792773 20538 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.775003 20614 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.795553 20614 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.796880 20614 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.797720 20538 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.799260 20538 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.801172 20611 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:11.801386 20596 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.801730 20611 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.799798 20613 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:11.802603 20533 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.802860 20613 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.803220 20533 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.798662 20614 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.797803 20610 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.776978 20322 ts_tablet_manager.cc:1403] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 2.058s user 0.022s sys 0.029s
I20260317 16:18:11.801977 20596 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.828136 20322 raft_consensus.cc:359] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.829175 20322 raft_consensus.cc:385] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:11.829531 20322 raft_consensus.cc:740] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:11.831015 20610 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.807606 20533 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.833801 20610 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.830420 20322 consensus_queue.cc:260] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.811926 20608 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:11.813257 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:11.839422 20120 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:11.839172 20608 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.841756 20322 ts_tablet_manager.cc:1434] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260317 16:18:11.843068 20322 tablet_bootstrap.cc:492] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:11.823086 20614 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.824424 20613 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.830226 20596 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.847504 20608 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.851075 20420 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:11.850942 20533 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.848047 20538 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.855480 20536 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.872097 20420 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.875756 20533 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.863597 20596 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.877188 20239 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:11.861585 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:11.879376 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 85604.1us
I20260317 16:18:11.879598 20536 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.885138 20608 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.885051 20590 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.888764 20527 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.889748 20608 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.886319 20464 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:11.902196 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:11.903276 20321 tablet_bootstrap.cc:492] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:11.903628 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:11.903294 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:11.883173 20596 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.904340 20123 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:11.904745 20119 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:11.903218 20608 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.905670 20608 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.905979 20596 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.906340 20536 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.906123 20608 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.898097 20527 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.908159 20608 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.908677 20608 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.909150 20527 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.909123 20608 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.900687 20464 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.903893 20321 ts_tablet_manager.cc:1403] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.838s user 0.021s sys 0.034s
I20260317 16:18:11.911684 20464 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.912206 20464 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.904366 20538 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:11.913286 20538 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:11.913735 20538 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:11.915943 20533 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:11.915970 20320 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 1.708s user 0.024s sys 0.002s
I20260317 16:18:11.916486 20608 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:11.916783 20320 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 1.709s user 0.025s sys 0.002s
I20260317 16:18:11.918051 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:11.919093 20320 tablet_bootstrap.cc:492] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:11.919569 20320 ts_tablet_manager.cc:1403] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 2.089s user 0.047s sys 0.004s
I20260317 16:18:11.916677 20239 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:11.920982 20239 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:11.921635 20533 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:11.923789 20533 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.924317 20533 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:11.913425 20321 raft_consensus.cc:359] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.912662 20464 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.925915 20321 raft_consensus.cc:385] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:11.896597 20611 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.922508 20536 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:11.887925 20120 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:11.926815 20321 raft_consensus.cc:740] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:11.925246 20538 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.935827 20321 consensus_queue.cc:260] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.937562 20538 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.938345 20424 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.939224 20424 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.938462 20321 ts_tablet_manager.cc:1434] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20260317 16:18:11.942015 20321 tablet_bootstrap.cc:492] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:11.924417 20322 tablet_bootstrap.cc:654] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:11.894281 20615 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.940642 20320 raft_consensus.cc:359] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.943411 20320 raft_consensus.cc:385] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:11.943650 20320 raft_consensus.cc:740] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:11.944106 20320 consensus_queue.cc:260] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.934432 20536 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.947065 20536 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:11.895895 20590 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.952165 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:11.951845 20615 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:11.952829 20119 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:11.953229 20590 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:11.953922 20617 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:11.954690 20615 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:11.954535 20617 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:11.957026 20320 ts_tablet_manager.cc:1434] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.037s user 0.005s sys 0.001s
I20260317 16:18:11.935094 20596 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.945933 20527 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.958129 20320 tablet_bootstrap.cc:492] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:11.966348 20464 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.969761 20617 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.957600 20596 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.978511 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:11.980197 20596 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.982466 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:11.982777 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:11.983242 20050 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:11.963104 20527 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:11.970731 20239 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:11.971427 20619 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:11.972492 20420 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:11.972769 20318 tablet_bootstrap.cc:654] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:11.973337 20424 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.983325 19944 rpcz_store.cc:275] Call kudu.master.MasterService.IsCreateTableDone from 127.0.0.1:46722 (request call id 35) took 1840 ms. Trace:
I20260317 16:18:11.970117 20615 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.989236 19944 rpcz_store.cc:276] 0317 16:18:10.143877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:10.217120 (+ 73243us) service_pool.cc:225] Handling call
0317 16:18:10.217477 (+ 357us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 16:18:11.982480 (+1765003us) catalog_manager.cc:2291] Verify if the table creation is in progress for test-workload [id=b34906c5f0e84d5b916910674fc7e084]
0317 16:18:11.983118 (+ 638us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1764731}
I20260317 16:18:11.990835 20618 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:11.991887 20321 tablet_bootstrap.cc:654] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:11.993471 20527 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:11.998375 20420 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.009817 20420 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:12.010257 20420 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.011420 20533 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:11.998545 20320 tablet_bootstrap.cc:654] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.014894 20533 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.000506 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.015682 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.001633 20619 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.017966 20619 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.002583 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.003424 20608 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.006291 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.014983 20272 raft_consensus.cc:1240] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
W20260317 16:18:11.999782 20162 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 2.008s (SENT) [suppressed 227 similar messages]
I20260317 16:18:11.984215 20119 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:12.020891 20447 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:12.023685 20424 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.023077 20610 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:12.022883 20613 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:12.051245 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.029677 20527 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.051756 20616 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.052101 20527 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.005448 20615 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.056102 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.050349 20610 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.057327 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.058036 20615 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.060019 20527 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.060518 20322 tablet_bootstrap.cc:492] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:12.060767 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.061131 20322 ts_tablet_manager.cc:1403] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.218s user 0.022s sys 0.013s
I20260317 16:18:12.058454 20610 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.061379 20049 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.043411 20617 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.062597 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.063088 20617 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.063300 20119 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:12.064678 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.065269 20617 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.049271 20596 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.066067 20596 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.026005 20608 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.067457 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.068027 20596 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.068373 20125 raft_consensus.cc:2393] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:12.066567 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.050511 20424 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.055023 20239 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:12.050413 20447 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.032021 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.084481 20121 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.053009 20613 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.087920 20613 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.069444 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.052287 20616 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.058462 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.082729 20424 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.083900 20447 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.031257 20536 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.089211 20322 raft_consensus.cc:359] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.070261 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.096318 20322 raft_consensus.cc:385] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.097586 20322 raft_consensus.cc:740] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.077596 20615 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.099272 20322 consensus_queue.cc:260] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.092103 20428 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:12.081007 20119 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.079022 20420 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.102370 20125 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:12.102434 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.102495 20623 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.110390 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:12.092070 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 92346.8us
I20260317 16:18:12.102308 20608 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.104163 20616 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.028051 20618 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.111032 20623 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.098837 20424 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.093528 20049 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:12.116283 20624 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.113324 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.117748 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:12.101456 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 80343.7us
I20260317 16:18:12.134657 20321 tablet_bootstrap.cc:492] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:12.129400 20624 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.111449 20195 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:12.124500 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.125262 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.125391 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.125563 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.115321 20428 raft_consensus.cc:697] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:12.127688 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.128851 20618 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.130661 20322 ts_tablet_manager.cc:1434] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.069s user 0.010s sys 0.001s
I20260317 16:18:12.132196 20628 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.132447 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.140848 20194 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.147964 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.148602 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.119196 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.135190 20321 ts_tablet_manager.cc:1403] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.193s user 0.024s sys 0.017s
I20260317 16:18:12.148996 20428 consensus_queue.cc:237] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.149518 20045 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:12.119359 20269 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.148730 20198 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.144096 20193 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:12.133813 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.135555 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.145447 20322 tablet_bootstrap.cc:492] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:12.135303 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.146462 20118 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.147488 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.148036 20199 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:12.152519 20321 raft_consensus.cc:359] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.154356 20420 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:12.152185 20220 debug-util.cc:398] Leaking SignalData structure 0x7b0800202860 after lost signal to thread 19982
I20260317 16:18:12.105885 20615 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.131734 20424 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.129638 20623 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.158167 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.172048 20192 raft_consensus.cc:2393] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:12.172199 20615 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.157861 20628 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.137437 20536 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.163327 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.173182 20198 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.164849 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.164994 20618 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.169163 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.169828 20196 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:12.176362 20618 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.165910 20321 raft_consensus.cc:385] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.171306 20624 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.175671 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.175930 20628 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.184814 20045 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:12.176062 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.186095 20117 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.186825 19947 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46692 (request call id 36) took 2657 ms. Trace:
I20260317 16:18:12.166679 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:12.177430 20321 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.161954 20119 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.179198 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.190521 20321 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.192283 20197 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.197947 20118 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.197816 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.192739 20239 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:12.192860 19945 catalog_manager.cc:5671] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:12.193976 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:12.195127 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:12.204226 20321 ts_tablet_manager.cc:1434] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20260317 16:18:12.246886 20016 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.259197 20321 tablet_bootstrap.cc:492] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:12.261196 20016 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.262283 20016 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.260471 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.260437 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.267324 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.267467 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.268074 20195 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.268658 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.269301 20536 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.269714 20536 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:12.269852 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.270107 20536 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.271943 20318 tablet_bootstrap.cc:492] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:12.272442 20318 ts_tablet_manager.cc:1403] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 2.033s user 0.014s sys 0.014s
I20260317 16:18:12.263411 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.273275 20016 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.207167 20628 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:12.275940 20320 tablet_bootstrap.cc:492] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:12.199431 20424 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.278311 20320 ts_tablet_manager.cc:1403] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.320s user 0.027s sys 0.011s
I20260317 16:18:12.281896 20320 raft_consensus.cc:359] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.283109 20622 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.283103 20320 raft_consensus.cc:385] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.283980 20622 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:12.284045 20320 raft_consensus.cc:740] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.284657 20622 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.203449 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:12.285151 20320 consensus_queue.cc:260] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.264402 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.198105 20618 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:12.264106 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 71584us
I20260317 16:18:12.238530 20118 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:12.301203 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.301714 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.303393 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.304886 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.259896 20590 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:12.306998 20590 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:12.307508 20590 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:12.308763 20220 debug-util.cc:398] Leaking SignalData structure 0x7b080029eda0 after lost signal to thread 20074
W20260317 16:18:12.309096 20220 debug-util.cc:398] Leaking SignalData structure 0x7b080035bb60 after lost signal to thread 20075
I20260317 16:18:12.309685 20536 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.263021 20124 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:12.272436 20619 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.299728 20318 raft_consensus.cc:359] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.312098 20318 raft_consensus.cc:385] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.297710 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.277920 20628 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:12.277323 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.234375 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.187208 19947 rpcz_store.cc:276] 0317 16:18:09.529320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:09.529483 (+ 163us) service_pool.cc:225] Handling call
0317 16:18:12.186698 (+2657215us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:09.731223 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:09.732047 (+ 824us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:09.732100 (+ 53us) write_op.cc:435] Acquired schema lock
0317 16:18:09.732127 (+ 27us) tablet.cc:661] Decoding operations
0317 16:18:09.732859 (+ 732us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:09.732935 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:09.732986 (+ 51us) tablet.cc:684] Acquiring locks for 3 operations
0317 16:18:09.733420 (+ 434us) tablet.cc:700] Row locks acquired
0317 16:18:09.733447 (+ 27us) write_op.cc:260] PREPARE: finished
0317 16:18:09.733648 (+ 201us) write_op.cc:270] Start()
0317 16:18:09.733816 (+ 168us) write_op.cc:276] Timestamp: P: 1773764289733586 usec, L: 0
0317 16:18:09.733847 (+ 31us) op_driver.cc:348] REPLICATION: starting
0317 16:18:09.734711 (+ 864us) log.cc:844] Serialized 1325 byte log entry
0317 16:18:09.791403 (+ 56692us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b1800346900. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea3cd4164d 00007fea3cd4394a 00007fea3cd438d8 00007fea3cd43858 00007fea3cd43813 00007fea3cd42a2c 00007fea3f142b26 00007fea3f13e278 00007fea3930c240
0317 16:18:09.791460 (+ 57us) op_driver.cc:464] REPLICATION: finished
0317 16:18:09.795409 (+ 3949us) write_op.cc:301] APPLY: starting
0317 16:18:09.795527 (+ 118us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:09.795712 (+ 185us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:09.795742 (+ 30us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:09.796128 (+ 386us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:09.796290 (+ 162us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0317 16:18:09.796348 (+ 58us) write_op.cc:312] APPLY: finished
0317 16:18:09.798533 (+ 2185us) log.cc:844] Serialized 34 byte log entry
0317 16:18:09.798721 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:09.806596 (+ 7875us) write_op.cc:454] Released schema lock
0317 16:18:09.816592 (+ 9996us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:09.821245 (+ 4653us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:09.821650 (+ 405us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1772207,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":3267,"apply.run_cpu_time_us":4015,"apply.run_wall_time_us":24346,"num_ops":3,"prepare.queue_time_us":717,"prepare.run_cpu_time_us":4211,"prepare.run_wall_time_us":9577,"replication_time_us":13042,"spinlock_wait_cycles":73495936}]]}
I20260317 16:18:12.312444 20318 raft_consensus.cc:740] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.304275 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.296571 20270 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.319718 20320 ts_tablet_manager.cc:1434] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.041s user 0.009s sys 0.000s
I20260317 16:18:12.319142 20318 consensus_queue.cc:260] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.274202 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:12.314455 20220 debug-util.cc:398] Leaking SignalData structure 0x7b08001530c0 after lost signal to thread 20149
I20260317 16:18:12.324944 20320 tablet_bootstrap.cc:492] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:12.298475 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.312669 20618 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.314747 20619 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:12.314064 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.326975 20618 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:12.343696 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.353565 20618 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.353785 20192 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:12.344163 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:12.326161 20220 debug-util.cc:398] Leaking SignalData structure 0x7b0800351c00 after lost signal to thread 20150
I20260317 16:18:12.344389 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.355088 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.344558 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.336319 20536 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.316152 20628 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.340874 20190 raft_consensus.cc:2393] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:12.340265 20533 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.339918 20420 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
W20260317 16:18:12.346302 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 73149.6us
I20260317 16:18:12.349722 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.349997 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.350752 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.362181 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.368643 20533 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.372802 20272 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.373812 20533 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.375082 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.368094 20420 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.351603 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.377399 20420 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.354437 20236 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.380528 20090 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:12.380787 20614 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.383049 20447 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.383435 20447 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.383824 20447 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.351728 20117 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:12.352735 20042 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:12.347031 20538 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.387805 20538 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.388235 20090 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:12.389788 20538 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.389700 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.346865 20619 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.390354 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.369029 20322 tablet_bootstrap.cc:654] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.391146 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.391266 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.391964 20046 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:12.392536 20612 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.392948 20612 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.393306 20612 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.390468 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:12.355937 20220 debug-util.cc:398] Leaking SignalData structure 0x7b08001da680 after lost signal to thread 20223
I20260317 16:18:12.398691 20318 ts_tablet_manager.cc:1434] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.126s user 0.007s sys 0.000s
I20260317 16:18:12.399869 20318 tablet_bootstrap.cc:492] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:12.405849 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.406008 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:12.397320 20220 debug-util.cc:398] Leaking SignalData structure 0x7b0800228d80 after lost signal to thread 20224
I20260317 16:18:12.407660 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.409289 20090 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:12.409667 20220 debug-util.cc:398] Leaking SignalData structure 0x7b0800333e80 after lost signal to thread 20297
I20260317 16:18:12.412246 20321 tablet_bootstrap.cc:654] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.399521 20271 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:12.411860 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.419525 20616 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.420118 20616 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.420538 20616 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.422711 20590 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.423240 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.424053 20049 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:12.425613 20590 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.429394 20165 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:12.430984 20424 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:12.431547 20424 raft_consensus.cc:697] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:12.432542 20424 consensus_queue.cc:237] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.434106 20616 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.434206 20320 tablet_bootstrap.cc:654] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.435107 20041 raft_consensus.cc:2393] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:12.402431 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.435241 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.401053 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.437180 20616 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.437538 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.439322 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.441321 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.441556 20622 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.441749 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.442060 20191 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
W20260317 16:18:12.399271 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 124889us
I20260317 16:18:12.443156 20162 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.444072 20624 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.444674 20624 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.444846 20015 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.445225 20624 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.443336 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.451771 20636 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:12.416182 20220 debug-util.cc:398] Leaking SignalData structure 0x7b080009c560 after lost signal to thread 20298
I20260317 16:18:12.456736 20590 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.463634 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:12.464133 20590 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.464434 20267 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.464645 20590 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.467555 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.467962 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.468915 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.469139 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.470038 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.470397 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.473397 20420 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.475991 20118 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.476191 20420 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.467628 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.478979 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.458456 20464 consensus_queue.cc:1048] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260317 16:18:12.479189 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.480168 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.480404 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.472155 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.459966 20616 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.485177 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.485278 20616 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.472329 20612 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.473331 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.471117 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.490424 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:12.463577 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.495376 20637 consensus_queue.cc:1059] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260317 16:18:12.496428 20637 raft_consensus.cc:3055] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.496474 20618 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.497272 20612 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.498241 20616 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.499130 20044 raft_consensus.cc:2393] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:12.479060 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.448982 20619 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.499187 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.490916 20628 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.480432 20270 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:12.496835 20637 raft_consensus.cc:740] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:12.491349 20464 consensus_queue.cc:1059] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260317 16:18:12.504585 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.506413 20119 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:12.508770 20447 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.487135 20015 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.510957 20264 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:12.511175 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:12.511412 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.498659 20618 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.511988 20420 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.501255 20636 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.509152 20267 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:12.526610 20636 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.527184 20636 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.512593 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.526450 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.522342 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:12.524559 20116 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.494168 20239 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.580216 20322 tablet_bootstrap.cc:492] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:12.580705 20322 ts_tablet_manager.cc:1403] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.435s user 0.018s sys 0.016s
I20260317 16:18:12.566936 20618 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.566586 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.582373 20420 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.581396 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:12.590611 20619 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.591265 20162 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.592532 20162 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.592624 20118 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:12.593537 20162 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:12.594983 20630 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:12.595715 20630 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.596046 20630 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.596374 20630 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.596482 20623 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.597273 20623 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:12.578984 20622 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:12.600893 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:12.587371 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 98632.8us
I20260317 16:18:12.603068 20447 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.604856 20636 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:12.605677 20641 log.cc:927] Time spent T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.068s user 0.000s sys 0.009s
I20260317 16:18:12.584816 20269 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:12.612341 20124 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.583305 20637 consensus_queue.cc:260] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.620918 20637 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.621909 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:12.623817 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:12.624572 20193 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.583606 20618 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.619203 20264 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:12.620815 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.625485 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.630203 20194 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:12.586603 20420 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:12.631134 20420 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.590615 20197 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:12.597726 20623 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.640383 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.643404 20239 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:12.644829 20538 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:12.645380 20538 raft_consensus.cc:697] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:12.645669 20239 leader_election.cc:343] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:12.646018 20636 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.646732 20538 consensus_queue.cc:237] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.645663 20624 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.650162 20199 raft_consensus.cc:1240] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.646451 20239 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:12.617486 20238 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:12.636989 20618 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.665192 20322 raft_consensus.cc:359] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.666996 20322 raft_consensus.cc:385] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.667625 20322 raft_consensus.cc:740] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.668581 20322 consensus_queue.cc:260] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.649009 20590 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.666301 20624 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.671437 20614 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:12.680207 20537 raft_consensus.cc:2764] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260317 16:18:12.687995 20322 ts_tablet_manager.cc:1434] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.107s user 0.007s sys 0.002s
I20260317 16:18:12.687374 20614 raft_consensus.cc:697] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:12.694751 20318 tablet_bootstrap.cc:654] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.700193 20590 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.672713 20420 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.702147 20618 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.702821 20420 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.699656 20614 consensus_queue.cc:237] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.704552 20618 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.691666 20322 tablet_bootstrap.cc:492] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:12.707557 20623 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.708129 20630 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.710371 20630 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.712352 20322 tablet_bootstrap.cc:654] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.712532 20321 tablet_bootstrap.cc:492] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:12.712986 20321 ts_tablet_manager.cc:1403] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.455s user 0.008s sys 0.015s
I20260317 16:18:12.715479 20320 tablet_bootstrap.cc:492] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:12.716178 20320 ts_tablet_manager.cc:1403] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.391s user 0.012s sys 0.009s
I20260317 16:18:12.715912 20321 raft_consensus.cc:359] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.716939 20321 raft_consensus.cc:385] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.717322 20321 raft_consensus.cc:740] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.718219 20321 consensus_queue.cc:260] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.726974 20321 ts_tablet_manager.cc:1434] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260317 16:18:12.710490 20623 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.729439 20320 raft_consensus.cc:359] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.730361 20320 raft_consensus.cc:385] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.730036 20321 tablet_bootstrap.cc:492] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:12.732390 20322 tablet_bootstrap.cc:492] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:12.733242 20322 ts_tablet_manager.cc:1403] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.004s
W20260317 16:18:12.563923 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 96889us
I20260317 16:18:12.780763 20015 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.585103 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.784554 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.580130 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.665477 20197 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:12.680394 20239 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:12.790213 20088 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:12.791668 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 118523us
W20260317 16:18:12.793519 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 195240us
I20260317 16:18:12.801050 20165 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.802313 20612 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:12.681103 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 64245.5us
I20260317 16:18:12.807041 20614 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:12.807947 20614 raft_consensus.cc:697] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:12.809994 20614 consensus_queue.cc:237] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.812350 20269 raft_consensus.cc:1240] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.817659 20165 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.819808 20088 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.820075 20238 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:12.730682 20320 raft_consensus.cc:740] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.823256 20590 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.824230 20623 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.827912 20091 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.832831 20537 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.833377 20045 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.834685 20590 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.835374 20590 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.835515 20238 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.836536 20608 raft_consensus.cc:3055] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.837000 20608 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:12.837002 20537 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.839833 20623 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.840503 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.838493 20608 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.842058 20608 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.842254 20318 tablet_bootstrap.cc:492] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:12.843247 20318 ts_tablet_manager.cc:1403] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.444s user 0.017s sys 0.004s
I20260317 16:18:12.848009 20616 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:12.852010 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.852311 20045 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.843149 20165 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.854871 20049 raft_consensus.cc:1240] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.855432 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.856283 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.856596 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.857146 20050 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.857647 20048 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:12.857420 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:12.858690 20042 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.859503 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:12.862414 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:12.855472 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.871594 20321 tablet_bootstrap.cc:654] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:12.737517 20322 raft_consensus.cc:359] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.874146 20628 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:12.876417 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.882018 20637 consensus_queue.cc:1048] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:12.884110 20320 consensus_queue.cc:260] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.886528 20193 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:12.887401 20623 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:12.890014 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 109706us
I20260317 16:18:12.890969 20447 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:12.889176 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 69587.2us
I20260317 16:18:12.874193 20322 raft_consensus.cc:385] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:12.894671 20322 raft_consensus.cc:740] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:12.895790 20322 consensus_queue.cc:260] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.896797 20015 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:12.899260 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:12.899561 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:12.899878 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:12.901705 20637 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.902944 20116 raft_consensus.cc:1240] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.903472 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:12.905529 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.907478 20015 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.909127 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.910794 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.911046 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.911551 20636 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.912019 20636 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:12.912451 20636 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.912705 20120 raft_consensus.cc:1240] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.912760 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:12.918295 20420 consensus_queue.cc:1059] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:12.919512 20623 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.919942 20420 raft_consensus.cc:3055] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.918829 20015 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:12.920392 20420 raft_consensus.cc:740] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:12.921639 20622 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.922025 20420 consensus_queue.cc:260] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.924026 20420 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.924362 20623 consensus_queue.cc:1059] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:12.925757 20615 raft_consensus.cc:3055] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.926142 20615 raft_consensus.cc:740] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:12.922441 20015 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:12.928880 20622 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:12.929315 20622 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:12.929723 20622 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.932039 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:12.933089 20117 raft_consensus.cc:1240] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.935423 20264 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.935671 20537 consensus_queue.cc:1059] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:12.937043 20428 raft_consensus.cc:3055] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.937515 20428 raft_consensus.cc:740] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:12.939862 20630 consensus_queue.cc:1059] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:12.938728 20428 consensus_queue.cc:260] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.942649 20630 consensus_queue.cc:1059] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260317 16:18:12.948052 20630 raft_consensus.cc:3055] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.948485 20630 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:12.949844 20630 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.943266 20624 raft_consensus.cc:3055] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.950992 20624 raft_consensus.cc:740] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:12.952379 20624 consensus_queue.cc:260] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:12.953011 19945 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46676 (request call id 39) took 3473 ms. Trace:
I20260317 16:18:12.954183 20624 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.927933 20615 consensus_queue.cc:260] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:12.950124 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 56307.6us
I20260317 16:18:12.944957 20428 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.962181 20615 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.963284 20320 ts_tablet_manager.cc:1434] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.247s user 0.011s sys 0.000s
I20260317 16:18:12.964277 20236 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.929517 20015 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:12.970278 20239 leader_election.cc:343] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:12.970373 20637 consensus_queue.cc:1059] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260317 16:18:12.972544 20614 consensus_queue.cc:1048] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:12.973856 20637 raft_consensus.cc:3055] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:12.974241 20637 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:12.974709 20088 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:12.975780 20612 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:12.975288 20637 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:12.978034 20637 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.978140 20088 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.979176 20614 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:12.979353 20616 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:12.980264 20320 tablet_bootstrap.cc:492] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:12.981315 20614 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:12.981873 20614 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.983493 20050 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:12.984102 20614 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:12.984462 20047 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:12.987015 20124 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:12.987332 20590 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.989574 20590 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:12.990581 20630 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:12.984493 20236 leader_election.cc:343] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:12.995254 20236 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:12.995401 20636 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:12.996182 20614 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:13.002985 20622 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.005564 20622 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:13.010943 20636 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:13.021046 20322 ts_tablet_manager.cc:1434] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.287s user 0.009s sys 0.004s
W20260317 16:18:13.023311 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 44124.3us
I20260317 16:18:13.023216 20614 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260317 16:18:13.027544 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 98184.2us
I20260317 16:18:13.023280 20537 consensus_queue.cc:1059] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260317 16:18:13.029997 20537 raft_consensus.cc:3055] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.030387 20537 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.033030 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.033443 20120 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:13.034065 20193 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:13.035177 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.036247 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.037416 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.040014 20124 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.042212 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.042889 20537 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.039949 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:13.048513 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 205685us
I20260317 16:18:13.049818 20238 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:13.047179 20537 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.050886 20614 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:13.055040 20614 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:13.055439 20614 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.020320 20636 consensus_queue.cc:1059] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260317 16:18:13.016227 20616 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.062609 20320 tablet_bootstrap.cc:654] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.050513 20165 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.066195 20165 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.052065 20238 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.053714 20322 tablet_bootstrap.cc:492] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:13.053445 20318 raft_consensus.cc:359] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.054473 20042 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:13.072690 20318 raft_consensus.cc:385] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.074847 20318 raft_consensus.cc:740] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.080812 20621 raft_consensus.cc:3055] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.081228 20621 raft_consensus.cc:740] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.082448 20621 consensus_queue.cc:260] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.084195 20621 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.079145 20318 consensus_queue.cc:260] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:13.086009 20162 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:13.088718 20322 tablet_bootstrap.cc:654] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.089231 19946 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46702 (request call id 34) took 3373 ms. Trace:
I20260317 16:18:13.089713 19946 rpcz_store.cc:276] 0317 16:18:09.716018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:09.716228 (+ 210us) service_pool.cc:225] Handling call
0317 16:18:13.089036 (+3372808us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20260317 16:18:13.100351 20047 raft_consensus.cc:1240] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.102032 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:13.104003 20043 raft_consensus.cc:1240] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.104310 20321 tablet_bootstrap.cc:492] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:13.104971 20321 ts_tablet_manager.cc:1403] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.375s user 0.038s sys 0.022s
I20260317 16:18:13.106426 20196 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:13.107007 19948 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46714 (request call id 37) took 3473 ms. Trace:
I20260317 16:18:13.108250 20318 ts_tablet_manager.cc:1434] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.264s user 0.010s sys 0.000s
I20260317 16:18:13.079452 20622 raft_consensus.cc:2749] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:13.111310 20318 tablet_bootstrap.cc:492] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:13.108598 19948 rpcz_store.cc:276] 0317 16:18:09.624702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:09.626024 (+ 1322us) service_pool.cc:225] Handling call
0317 16:18:13.097825 (+3471801us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10913,"reported_tablets":9}
I20260317 16:18:13.119311 20046 raft_consensus.cc:1240] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:13.111346 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:13.117076 20614 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.085175 20630 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.098547 20420 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.122690 20624 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.133401 20045 raft_consensus.cc:1240] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.134130 20538 consensus_queue.cc:1059] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:13.135167 20536 consensus_queue.cc:1059] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:13.145364 20533 raft_consensus.cc:3055] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.145763 20614 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:13.146236 20533 raft_consensus.cc:740] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.134940 20428 consensus_queue.cc:1059] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:13.147325 20428 raft_consensus.cc:3055] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.147701 20428 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.147451 20533 consensus_queue.cc:260] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.149140 20533 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.148588 20428 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.150350 20428 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.151186 20047 raft_consensus.cc:1240] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.139920 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.140528 20162 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:13.142721 20043 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:13.125097 20199 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:13.154654 20630 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.154744 20162 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:13.143689 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.143806 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:13.144064 20088 leader_election.cc:343] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.146409 20536 raft_consensus.cc:3055] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.156903 20615 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.157034 20536 raft_consensus.cc:740] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.157550 20615 raft_consensus.cc:697] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:13.158203 20536 consensus_queue.cc:260] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.143703 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.159590 20536 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.158596 20615 consensus_queue.cc:237] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:12.953512 19945 rpcz_store.cc:276] 0317 16:18:09.479403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:09.480696 (+ 1293us) service_pool.cc:225] Handling call
0317 16:18:12.952732 (+3472036us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:12.297193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:12.419136 (+121943us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:12.419178 (+ 42us) write_op.cc:435] Acquired schema lock
0317 16:18:12.419205 (+ 27us) tablet.cc:661] Decoding operations
0317 16:18:12.419610 (+ 405us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:12.419691 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:12.419728 (+ 37us) tablet.cc:684] Acquiring locks for 1 operations
0317 16:18:12.419929 (+ 201us) tablet.cc:700] Row locks acquired
0317 16:18:12.419949 (+ 20us) write_op.cc:260] PREPARE: finished
0317 16:18:12.420129 (+ 180us) write_op.cc:270] Start()
0317 16:18:12.420252 (+ 123us) write_op.cc:276] Timestamp: P: 1773764292420072 usec, L: 0
0317 16:18:12.420276 (+ 24us) op_driver.cc:348] REPLICATION: starting
0317 16:18:12.421152 (+ 876us) log.cc:844] Serialized 543 byte log entry
0317 16:18:12.728651 (+307499us) op_driver.cc:464] REPLICATION: finished
0317 16:18:12.739152 (+ 10501us) write_op.cc:301] APPLY: starting
0317 16:18:12.739243 (+ 91us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:12.739350 (+ 107us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:12.739481 (+ 131us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:12.739755 (+ 274us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:12.739929 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 16:18:12.739988 (+ 59us) write_op.cc:312] APPLY: finished
0317 16:18:12.740184 (+ 196us) log.cc:844] Serialized 22 byte log entry
0317 16:18:12.740343 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:12.740430 (+ 87us) write_op.cc:454] Released schema lock
0317 16:18:12.740470 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:12.740780 (+ 310us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:12.740810 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2402224,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":10243,"apply.run_cpu_time_us":1635,"apply.run_wall_time_us":1635,"num_ops":1,"prepare.queue_time_us":21016,"prepare.run_cpu_time_us":14035,"prepare.run_wall_time_us":214200,"raft.queue_time_us":70015,"raft.run_cpu_time_us":8630,"raft.run_wall_time_us":11292,"replication_time_us":307451,"thread_start_us":166330,"threads_started":4,"wal-append.queue_time_us":151087}]]}
W20260317 16:18:13.154161 20163 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:13.155303 20630 raft_consensus.cc:697] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:13.165881 20630 consensus_queue.cc:237] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.156227 20628 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.166477 20191 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:13.167090 20628 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.151165 20050 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:13.152367 20321 raft_consensus.cc:359] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.169142 20321 raft_consensus.cc:385] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.169251 20628 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:13.169466 20321 raft_consensus.cc:740] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.170176 20321 consensus_queue.cc:260] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.172891 20321 ts_tablet_manager.cc:1434] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.067s user 0.007s sys 0.002s
I20260317 16:18:13.174115 20321 tablet_bootstrap.cc:492] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:13.174784 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:13.123656 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 57626us
W20260317 16:18:13.119896 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 66939.4us
I20260317 16:18:13.173887 20015 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.179714 20622 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.180281 20015 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.181563 20622 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.185956 20320 tablet_bootstrap.cc:492] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
W20260317 16:18:13.186915 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 54500.8us
I20260317 16:18:13.187214 20320 ts_tablet_manager.cc:1403] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.207s user 0.015s sys 0.015s
I20260317 16:18:13.188985 20321 tablet_bootstrap.cc:654] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.190512 20320 raft_consensus.cc:359] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.182386 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:13.195917 20318 tablet_bootstrap.cc:654] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.192687 20320 raft_consensus.cc:385] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.197286 20320 raft_consensus.cc:740] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.198195 20165 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.200284 20322 tablet_bootstrap.cc:492] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:13.200914 20322 ts_tablet_manager.cc:1403] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.148s user 0.016s sys 0.013s
I20260317 16:18:13.200017 20624 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.203183 20165 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.205852 20165 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.207134 20623 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.207979 20623 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.209905 20165 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.211629 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:13.213184 20165 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.218161 20320 consensus_queue.cc:260] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.225904 20533 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.227165 20322 raft_consensus.cc:359] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.264467 20322 raft_consensus.cc:385] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.264839 20322 raft_consensus.cc:740] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
W20260317 16:18:13.217370 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 82671us
I20260317 16:18:13.182487 20015 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:13.225633 20623 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:13.225798 20624 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.262140 20628 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.267110 20322 consensus_queue.cc:260] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:13.263621 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 50549.4us
W20260317 16:18:13.269086 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 86701.3us
I20260317 16:18:13.270579 20630 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.268921 20621 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.277863 20621 raft_consensus.cc:697] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.279093 20621 consensus_queue.cc:237] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.272604 20628 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.287498 20322 ts_tablet_manager.cc:1434] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.086s user 0.021s sys 0.005s
I20260317 16:18:13.288812 20322 tablet_bootstrap.cc:492] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
W20260317 16:18:13.292976 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 105014us
W20260317 16:18:13.299042 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 118973us
W20260317 16:18:13.300143 20238 leader_election.cc:343] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.301163 20238 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:13.302461 20536 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:13.303542 20165 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.304698 20322 tablet_bootstrap.cc:654] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.305042 20661 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.305271 20165 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.305963 20616 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.306555 20616 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.308838 20616 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:13.309054 20661 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.310042 20428 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.310510 20428 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.313330 20428 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:13.315842 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.317394 20622 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.318008 20016 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:13.320799 20622 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:13.321427 20622 raft_consensus.cc:697] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.322715 20628 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:13.319599 20015 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:13.322431 20622 consensus_queue.cc:237] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.325819 20090 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.269454 20623 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:13.338020 20321 tablet_bootstrap.cc:492] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:13.338583 20321 ts_tablet_manager.cc:1403] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.165s user 0.016s sys 0.024s
I20260317 16:18:13.330977 20623 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.341821 20464 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.342935 20236 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:13.342599 20464 raft_consensus.cc:697] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.344254 20264 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.330658 20637 consensus_queue.cc:1059] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
W20260317 16:18:13.346378 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:13.341909 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:13.347895 20637 raft_consensus.cc:3055] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.348309 20637 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.342983 20428 consensus_queue.cc:1059] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260317 16:18:13.349269 20637 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.349380 20613 consensus_queue.cc:1059] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:18:13.350219 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:13.350934 20616 raft_consensus.cc:3055] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.350847 20613 consensus_queue.cc:1059] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:13.351364 20616 raft_consensus.cc:740] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:13.350659 20637 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.351926 20613 raft_consensus.cc:3055] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.352348 20613 raft_consensus.cc:740] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:13.352421 20616 consensus_queue.cc:260] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.353736 20613 consensus_queue.cc:260] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.346127 20464 consensus_queue.cc:237] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.361497 20613 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.343163 20612 raft_consensus.cc:2804] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:13.362988 20612 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:13.363055 20428 raft_consensus.cc:3055] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.363399 20612 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.363540 20428 raft_consensus.cc:740] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.364420 20428 consensus_queue.cc:260] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.351368 20236 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:13.366195 20428 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.366262 20538 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.366829 20536 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:13.366848 20538 raft_consensus.cc:697] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.367257 20536 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:13.367784 20536 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.368201 20538 consensus_queue.cc:237] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.371544 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.372463 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:13.372954 20120 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.373340 20117 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:13.375069 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:13.373068 20090 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.377032 20447 raft_consensus.cc:2749] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.378155 20321 raft_consensus.cc:359] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.378909 20321 raft_consensus.cc:385] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.379241 20321 raft_consensus.cc:740] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.380198 20090 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:13.386052 20321 consensus_queue.cc:260] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.388899 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:13.389298 20050 raft_consensus.cc:1275] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260317 16:18:13.389671 20119 raft_consensus.cc:1275] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:13.389680 20047 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:13.389775 20050 pending_rounds.cc:85] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Aborting all ops after (but not including) 0
I20260317 16:18:13.390446 20050 pending_rounds.cc:107] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260317 16:18:13.390741 20050 raft_consensus.cc:2889] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:13.392002 20090 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.392876 20321 ts_tablet_manager.cc:1434] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20260317 16:18:13.393045 20449 raft_consensus.cc:2804] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:13.353973 20616 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.398906 20485 consensus_queue.cc:1059] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260317 16:18:13.399509 20016 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.401042 20016 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.414832 20608 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.416703 20322 tablet_bootstrap.cc:492] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:13.417197 20322 ts_tablet_manager.cc:1403] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.129s user 0.017s sys 0.003s
I20260317 16:18:13.414558 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:13.421563 20321 tablet_bootstrap.cc:492] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:13.424274 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.425668 20264 raft_consensus.cc:2410] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260317 16:18:13.413498 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:13.437081 20320 ts_tablet_manager.cc:1434] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.249s user 0.005s sys 0.005s
I20260317 16:18:13.407316 20447 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:13.438099 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:13.439230 20050 raft_consensus.cc:1275] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:13.409371 20622 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.439419 20236 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:13.453620 20420 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:13.454067 20420 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:13.454439 20420 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.457641 20614 consensus_queue.cc:1048] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 16:18:13.432916 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 113428us
I20260317 16:18:13.459132 20533 consensus_queue.cc:1048] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:13.460028 20015 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:13.465528 20090 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:13.468279 20322 raft_consensus.cc:359] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.469070 20322 raft_consensus.cc:385] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.469348 20322 raft_consensus.cc:740] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.470062 20322 consensus_queue.cc:260] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.470947 20612 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.473287 20612 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:13.475931 20318 tablet_bootstrap.cc:492] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:13.476413 20318 ts_tablet_manager.cc:1403] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.365s user 0.008s sys 0.014s
W20260317 16:18:13.477227 20239 leader_election.cc:343] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.481588 20318 raft_consensus.cc:359] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.483232 20608 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.489579 20608 raft_consensus.cc:697] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.490572 20608 consensus_queue.cc:237] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.485379 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:13.492048 20322 ts_tablet_manager.cc:1434] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.075s user 0.008s sys 0.000s
I20260317 16:18:13.492353 20117 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:13.486994 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:13.493268 20322 tablet_bootstrap.cc:492] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:13.494170 20318 raft_consensus.cc:385] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.488121 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:13.488588 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:13.491791 20614 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:13.487212 20536 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.485844 20537 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:13.494630 20318 raft_consensus.cc:740] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.509322 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:13.510069 20199 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:13.510291 20196 raft_consensus.cc:1240] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.510517 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.511219 20198 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.512434 20272 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:13.514556 20318 consensus_queue.cc:260] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.506497 20672 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.513397 20265 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:13.518481 20536 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:13.523192 20669 consensus_queue.cc:1048] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:13.526309 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:13.512260 20420 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.528028 20320 tablet_bootstrap.cc:492] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:13.514472 20449 raft_consensus.cc:697] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:13.522262 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:13.543921 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 102641us
I20260317 16:18:13.536568 20043 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.544857 20447 raft_consensus.cc:3055] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.545265 20447 raft_consensus.cc:740] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:13.549654 20447 consensus_queue.cc:260] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.509899 19946 catalog_manager.cc:5671] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:13.541008 20672 raft_consensus.cc:697] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:13.556816 20318 ts_tablet_manager.cc:1434] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.080s user 0.008s sys 0.000s
I20260317 16:18:13.562402 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:13.558355 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:13.567123 20045 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.553944 20628 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:13.561125 19946 catalog_manager.cc:5671] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 1 to 2, leader changed from 9a3afb098d6a45f0b405566771a1157b (127.16.123.193) to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:13.571393 20449 consensus_queue.cc:237] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.572003 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:13.574543 19946 catalog_manager.cc:5671] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 1 to 2, leader changed from 9a3afb098d6a45f0b405566771a1157b (127.16.123.193) to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:13.559746 20616 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:13.578744 20616 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.581701 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.584460 20119 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:13.585067 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.586982 20046 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.577034 19946 catalog_manager.cc:5671] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 reported cstate change: term changed from 1 to 2, leader changed from 9a3afb098d6a45f0b405566771a1157b (127.16.123.193) to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:13.591815 20318 tablet_bootstrap.cc:492] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
W20260317 16:18:13.568598 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 44249us
I20260317 16:18:13.584380 20447 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.597657 20048 raft_consensus.cc:1240] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.564781 20091 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:13.599877 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 44911.1us
I20260317 16:18:13.602159 20613 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.600960 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:13.603916 20197 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:13.596235 20420 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:13.607818 20322 tablet_bootstrap.cc:654] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.608661 20321 tablet_bootstrap.cc:654] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.610267 20198 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:13.605194 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.604763 20091 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.590201 20616 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:13.625370 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.626269 20268 raft_consensus.cc:2376] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:13.627461 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:13.628208 20267 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:13.628208 20420 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.629853 20420 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.593216 20269 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:13.615330 20041 raft_consensus.cc:1240] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.633137 20091 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.621784 20613 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.639182 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:13.639061 20536 consensus_queue.cc:1059] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:18:13.640326 20676 log.cc:927] Time spent T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260317 16:18:13.652439 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:13.653343 20048 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:13.595101 20672 consensus_queue.cc:237] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.645318 20485 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.645469 20165 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:13.631770 20623 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.658002 20165 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:13.658319 20623 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:13.649361 20318 tablet_bootstrap.cc:654] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.658995 20191 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:13.659166 20669 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:13.659636 20669 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:13.659978 20669 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.654585 20420 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:13.661218 20673 log.cc:927] Time spent T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260317 16:18:13.659044 20623 consensus_queue.cc:1048] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260317 16:18:13.647362 20091 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:13.652515 20536 raft_consensus.cc:3055] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.644974 20320 tablet_bootstrap.cc:654] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:13.654194 20537 consensus_queue.cc:1059] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:13.665880 20536 raft_consensus.cc:740] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.674129 20536 consensus_queue.cc:260] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:13.675397 20536 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.658989 20661 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260317 16:18:13.670007 20165 leader_election.cc:343] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.681067 20420 raft_consensus.cc:3055] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:13.681205 20613 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.673949 20088 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:13.692446 20016 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:13.697502 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:13.698707 20043 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:13.700762 20124 raft_consensus.cc:1275] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:13.696174 20608 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:13.686419 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:13.705721 20322 tablet_bootstrap.cc:492] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:13.706486 20322 ts_tablet_manager.cc:1403] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.213s user 0.010s sys 0.008s
W20260317 16:18:13.710912 20239 leader_election.cc:343] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.712888 20267 raft_consensus.cc:1275] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:13.688971 20420 raft_consensus.cc:740] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.710304 20165 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:13.707595 20612 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:13.704774 20608 raft_consensus.cc:697] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:13.713619 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:13.748049 20045 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
W20260317 16:18:13.754953 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:13.756206 20318 tablet_bootstrap.cc:492] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:13.756705 20318 ts_tablet_manager.cc:1403] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.179s user 0.013s sys 0.001s
I20260317 16:18:13.707167 20090 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:13.759022 20628 consensus_queue.cc:1048] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260317 16:18:13.759608 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 63121.5us
I20260317 16:18:13.762117 20090 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:13.727105 20538 consensus_queue.cc:1048] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
W20260317 16:18:13.765164 20238 leader_election.cc:343] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.767030 20238 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:13.767885 20116 raft_consensus.cc:1275] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:13.730399 20661 raft_consensus.cc:2804] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:13.768033 20536 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:13.736112 20612 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:13.768627 20661 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:13.786019 20318 raft_consensus.cc:359] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.791287 20318 raft_consensus.cc:385] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.791718 20318 raft_consensus.cc:740] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.869398 20318 consensus_queue.cc:260] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:13.786146 20613 raft_consensus.cc:2804] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:13.776926 20265 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:13.722947 20236 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:13.790059 20612 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.887216 20321 tablet_bootstrap.cc:492] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:13.887912 20321 ts_tablet_manager.cc:1403] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.494s user 0.013s sys 0.006s
I20260317 16:18:13.777534 20622 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20260317 16:18:13.799191 20661 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.874430 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:13.770517 20420 consensus_queue.cc:260] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:13.914587 20318 ts_tablet_manager.cc:1434] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.158s user 0.030s sys 0.057s
W20260317 16:18:13.891191 20238 leader_election.cc:343] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:13.905699 20613 raft_consensus.cc:697] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:13.891844 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 104146us
I20260317 16:18:13.905473 20536 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
W20260317 16:18:13.904295 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 189337us
I20260317 16:18:13.914048 20045 raft_consensus.cc:1275] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:13.936522 20191 raft_consensus.cc:1275] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:13.912485 20272 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:13.929025 20674 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.924373 20677 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:13.960141 20320 tablet_bootstrap.cc:492] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:13.900605 20608 consensus_queue.cc:237] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.960987 20320 ts_tablet_manager.cc:1403] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.462s user 0.049s sys 0.034s
I20260317 16:18:13.925209 20613 consensus_queue.cc:237] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:13.962916 20637 consensus_queue.cc:1048] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:13.963526 20675 consensus_queue.cc:1059] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260317 16:18:13.946529 20670 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:13.965862 20420 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:13.949546 20536 raft_consensus.cc:697] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:13.989259 20536 consensus_queue.cc:237] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:13.997524 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 229295us
W20260317 16:18:13.982916 20680 log.cc:927] Time spent T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.188s user 0.000s sys 0.002s
I20260317 16:18:13.961242 20677 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:13.985831 20322 raft_consensus.cc:359] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:14.019949 20322 raft_consensus.cc:385] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:13.987962 20050 raft_consensus.cc:1275] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.020318 20322 raft_consensus.cc:740] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:13.998075 20046 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
W20260317 16:18:13.985127 20236 leader_election.cc:343] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.021241 20322 consensus_queue.cc:260] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:14.081506 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 54565us
I20260317 16:18:14.082386 20091 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:14.008394 20669 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.095368 20321 raft_consensus.cc:359] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.095666 20612 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:14.096160 20321 raft_consensus.cc:385] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:14.096378 20669 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:14.096479 20321 raft_consensus.cc:740] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
W20260317 16:18:14.083992 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 98902.9us
I20260317 16:18:14.097968 20321 consensus_queue.cc:260] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.098694 20272 raft_consensus.cc:1275] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
W20260317 16:18:14.098934 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 93570.3us
I20260317 16:18:14.099182 20272 pending_rounds.cc:85] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Aborting all ops after (but not including) 0
I20260317 16:18:14.099486 20272 pending_rounds.cc:107] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260317 16:18:14.099756 20272 raft_consensus.cc:2889] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:14.100013 20612 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:14.100445 20318 tablet_bootstrap.cc:492] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:14.009420 20670 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.086838 20198 raft_consensus.cc:1275] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.086838 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:14.089701 20320 raft_consensus.cc:359] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:14.093389 20449 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:14.014959 20047 raft_consensus.cc:1275] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.105705 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:14.111493 20320 raft_consensus.cc:385] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:14.119225 20320 raft_consensus.cc:740] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:14.022981 20677 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:13.979446 20674 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:14.020273 20623 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.091s
I20260317 16:18:14.128007 20669 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:14.133936 20320 consensus_queue.cc:260] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:14.109915 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:14.136762 20637 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:14.120724 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 97010.1us
I20260317 16:18:14.121513 20449 raft_consensus.cc:697] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:14.116462 20116 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:14.138239 20621 consensus_queue.cc:1048] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:14.145720 20322 ts_tablet_manager.cc:1434] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.439s user 0.021s sys 0.039s
W20260317 16:18:14.143208 20686 log.cc:927] Time spent T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.358s user 0.006s sys 0.000s
I20260317 16:18:14.163403 20321 ts_tablet_manager.cc:1434] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.275s user 0.007s sys 0.002s
I20260317 16:18:14.158905 20623 consensus_queue.cc:1048] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.012s
I20260317 16:18:14.171808 20191 raft_consensus.cc:1275] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.152910 20637 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.174839 20637 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:14.159669 20120 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:14.157543 20449 consensus_queue.cc:237] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.179913 20670 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:14.161289 20464 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260317 16:18:14.188002 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:14.153008 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 47412.3us
I20260317 16:18:14.192464 20537 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:14.196805 20661 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.198987 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:14.200862 20674 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:14.203027 20320 ts_tablet_manager.cc:1434] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.237s user 0.004s sys 0.004s
W20260317 16:18:14.209523 20238 leader_election.cc:343] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:14.209627 20016 leader_election.cc:343] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.210467 20238 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:14.230145 20321 tablet_bootstrap.cc:492] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:14.164758 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 66098.2us
I20260317 16:18:14.245972 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:14.247486 20045 raft_consensus.cc:2393] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:14.263345 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:14.267432 20119 raft_consensus.cc:1275] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.282886 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:14.206903 20537 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:14.283876 20696 log.cc:927] Time spent T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
W20260317 16:18:14.285393 20015 leader_election.cc:343] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.285759 20236 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:14.229369 20628 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
W20260317 16:18:14.283864 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 70598us
I20260317 16:18:14.287487 20636 consensus_queue.cc:1048] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:14.211472 20464 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
W20260317 16:18:14.232080 20088 leader_election.cc:343] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:14.289991 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 60247.7us
I20260317 16:18:14.255678 20538 raft_consensus.cc:2749] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:14.285869 20537 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:14.300729 20050 raft_consensus.cc:1275] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.311195 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:14.312732 20320 tablet_bootstrap.cc:492] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
W20260317 16:18:14.313117 20238 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.308020 20537 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:14.329846 20318 tablet_bootstrap.cc:654] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:14.325577 20537 raft_consensus.cc:697] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:14.331610 20537 consensus_queue.cc:237] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:14.286804 20015 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:14.340914 20485 raft_consensus.cc:3055] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:14.341526 20485 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:14.374047 20320 tablet_bootstrap.cc:654] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:14.342594 20485 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.324107 20420 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260317 16:18:14.373193 20636 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:14.397063 20612 consensus_queue.cc:1048] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:14.399003 20116 raft_consensus.cc:1275] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.373041 20661 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:14.364779 20536 consensus_queue.cc:1048] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260317 16:18:14.405167 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:14.405536 20636 consensus_queue.cc:1048] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:14.409605 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:14.410118 20322 tablet_bootstrap.cc:492] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:14.409984 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:14.410516 20267 raft_consensus.cc:2393] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:14.410909 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:14.411038 20270 raft_consensus.cc:1275] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.411571 20271 raft_consensus.cc:1240] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:14.411845 20265 raft_consensus.cc:2393] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:14.414019 20091 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:14.385437 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 100100us
I20260317 16:18:14.444062 20046 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:14.445008 20677 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.458979 20699 consensus_queue.cc:1059] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260317 16:18:14.460196 20699 raft_consensus.cc:3055] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:14.460532 20699 raft_consensus.cc:740] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:14.461661 20321 tablet_bootstrap.cc:654] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:14.461464 20699 consensus_queue.cc:260] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.439716 20670 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:14.439128 20449 consensus_queue.cc:1048] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260317 16:18:14.462708 20699 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:14.498863 20088 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:14.446331 20536 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260317 16:18:14.491593 20485 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:14.522344 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 169659us
W20260317 16:18:14.524077 20165 leader_election.cc:343] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.534032 20196 raft_consensus.cc:1275] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.514179 20117 raft_consensus.cc:2393] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:14.454406 20674 consensus_queue.cc:1048] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260317 16:18:14.535329 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:14.566635 20123 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:14.539345 20616 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:14.568608 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:14.568894 20616 raft_consensus.cc:697] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:14.570513 20194 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:14.571075 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:14.572242 20268 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:14.536006 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:14.575387 20272 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:14.589445 20322 tablet_bootstrap.cc:654] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:14.591053 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:14.603046 20709 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:14.603932 20709 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:14.549810 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 64027.1us
I20260317 16:18:14.570209 20616 consensus_queue.cc:237] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:14.591322 20120 raft_consensus.cc:1275] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.633074 20321 tablet_bootstrap.cc:492] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:14.615440 20613 consensus_queue.cc:1048] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260317 16:18:14.635803 20321 ts_tablet_manager.cc:1403] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.414s user 0.012s sys 0.013s
I20260317 16:18:14.626345 20709 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:14.629252 20236 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.643564 20677 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:14.661751 20320 tablet_bootstrap.cc:492] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:14.664307 20320 ts_tablet_manager.cc:1403] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.352s user 0.013s sys 0.008s
I20260317 16:18:14.665649 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:14.669955 20090 leader_election.cc:343] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:14.657660 20236 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:14.665714 20637 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260317 16:18:14.707170 20318 tablet_bootstrap.cc:492] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:14.707872 20318 ts_tablet_manager.cc:1403] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.608s user 0.027s sys 0.027s
I20260317 16:18:14.712206 20538 raft_consensus.cc:2749] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:14.691453 20090 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:14.739399 20268 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
W20260317 16:18:14.730578 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 102263us
I20260317 16:18:14.748903 20165 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:14.716849 20700 consensus_queue.cc:1048] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.117s
W20260317 16:18:14.769272 20715 log.cc:927] Time spent T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260317 16:18:14.764835 20613 raft_consensus.cc:2749] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:14.773670 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 106461us
I20260317 16:18:14.809772 20623 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:14.810415 20623 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:14.811699 20701 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:14.812316 20538 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:14.813021 20538 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:14.816285 20614 consensus_queue.cc:1048] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:14.809475 20318 raft_consensus.cc:359] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.821177 20318 raft_consensus.cc:385] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:14.821547 20318 raft_consensus.cc:740] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:14.789906 20123 raft_consensus.cc:1275] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.822343 20318 consensus_queue.cc:260] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.834069 20713 raft_consensus.cc:2804] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:14.870875 20320 raft_consensus.cc:359] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.871910 20320 raft_consensus.cc:385] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:14.872257 20320 raft_consensus.cc:740] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:14.873090 20320 consensus_queue.cc:260] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.812378 20701 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:14.878365 20318 ts_tablet_manager.cc:1434] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.170s user 0.009s sys 0.000s
W20260317 16:18:14.873978 20727 log.cc:927] Time spent T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.157s user 0.000s sys 0.002s
I20260317 16:18:14.887924 20321 raft_consensus.cc:359] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.889020 20321 raft_consensus.cc:385] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:14.889377 20321 raft_consensus.cc:740] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:14.883051 20538 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:14.902642 20701 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:14.908726 20318 tablet_bootstrap.cc:492] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:14.892076 20321 consensus_queue.cc:260] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:14.881440 20614 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:14.923700 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:14.937021 20707 log.cc:927] Time spent T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.503s user 0.004s sys 0.000s
W20260317 16:18:14.941723 20716 log.cc:927] Time spent T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.357s user 0.003s sys 0.001s
I20260317 16:18:14.945897 20196 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:14.896831 20273 raft_consensus.cc:1275] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:14.897313 20613 consensus_queue.cc:1048] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.081s
I20260317 16:18:14.905172 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:14.906042 20670 raft_consensus.cc:2749] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:15.003569 20621 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.032718 20194 raft_consensus.cc:2393] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:15.049484 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:14.995347 20677 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.068769 20677 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:15.071863 20272 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:14.972982 20614 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.054411 20621 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.078680 20614 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.082814 20190 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:15.091150 20677 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.153077 20674 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.153811 20674 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:15.122607 20608 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.148416 20614 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:15.200834 20614 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:15.203964 20614 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.191457 20674 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.207631 20623 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.208702 20608 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:15.172123 20621 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.212670 20428 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:15.097229 20699 consensus_meta.cc:287] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.591s user 0.012s sys 0.002s
W20260317 16:18:15.156699 20485 consensus_meta.cc:287] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.625s user 0.014s sys 0.001s
W20260317 16:18:15.220825 20692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.002s sys 0.034s
W20260317 16:18:15.225010 20191 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.019s sys 0.043s
I20260317 16:18:15.348031 20318 tablet_bootstrap.cc:654] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:15.338294 20621 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:15.287358 20708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.014s sys 0.059s
W20260317 16:18:15.349022 20013 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.016s sys 0.020s
W20260317 16:18:15.267567 20265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.017s sys 0.024s
W20260317 16:18:15.335516 20271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.026s sys 0.041s
W20260317 16:18:15.338327 20191 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.019s sys 0.043s
I20260317 16:18:15.339003 20428 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:15.265429 20718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.032s sys 0.009s
W20260317 16:18:15.337064 20692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.002s sys 0.035s
W20260317 16:18:15.354512 20265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.017s sys 0.027s
I20260317 16:18:15.347170 20485 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.393455 20485 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:15.396934 20190 raft_consensus.cc:1275] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260317 16:18:15.352355 20708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s user 0.014s sys 0.060s
W20260317 16:18:15.352231 20013 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s user 0.016s sys 0.021s
W20260317 16:18:15.354833 20271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.027s sys 0.041s
I20260317 16:18:15.391140 20608 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:15.412212 20740 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.427840 20740 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:15.349538 20621 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.433086 20621 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.421542 20730 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:15.384441 20718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.032s sys 0.010s
I20260317 16:18:15.438262 20730 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.441417 20730 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.419811 20485 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.427174 20428 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.451321 20740 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.471186 20196 raft_consensus.cc:1275] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:15.529347 20485 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.531490 20733 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.532003 20485 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:15.532326 20733 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.534260 20718 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.536098 20718 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.542112 20485 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.547559 20733 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.534657 20628 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.539366 20740 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.576011 20621 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.597673 20740 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:15.555511 20628 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:15.627705 20322 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.033s sys 0.048s
W20260317 16:18:15.628223 20322 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.814s user 0.033s sys 0.048s
W20260317 16:18:15.627696 20042 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.039s sys 0.060s
I20260317 16:18:15.616380 20485 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.624610 20674 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.619578 20718 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:15.624897 20713 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.006s sys 0.011s
I20260317 16:18:15.645449 20674 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:15.626266 20699 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:15.635309 20239 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.031s sys 0.039s
I20260317 16:18:15.617784 20621 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:15.658054 20536 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.006s sys 0.017s
W20260317 16:18:15.659144 20238 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.021s sys 0.022s
W20260317 16:18:15.653518 20713 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.006s sys 0.012s
I20260317 16:18:15.644062 20485 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:15.663936 20239 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s user 0.032s sys 0.039s
W20260317 16:18:15.634132 20042 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.039s sys 0.061s
W20260317 16:18:15.656411 20321 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.009s sys 0.011s
W20260317 16:18:15.730842 20321 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.821s user 0.011s sys 0.012s
W20260317 16:18:15.669837 20236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.010s sys 0.061s
W20260317 16:18:15.673669 20320 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.034s sys 0.037s
W20260317 16:18:15.683441 20700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.013s sys 0.021s
W20260317 16:18:15.731668 20320 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.857s user 0.035s sys 0.038s
I20260317 16:18:15.662189 20699 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:15.682236 20091 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.011s sys 0.051s
W20260317 16:18:15.734668 20091 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.764s user 0.012s sys 0.051s
W20260317 16:18:15.683100 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:15.670944 20670 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
W20260317 16:18:15.680989 20120 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.012s sys 0.052s
W20260317 16:18:15.723599 20165 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.008s sys 0.012s
W20260317 16:18:15.738252 20165 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s user 0.008s sys 0.012s
W20260317 16:18:15.726259 20238 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.863s user 0.022s sys 0.023s
I20260317 16:18:15.728832 20713 raft_consensus.cc:697] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:15.738858 20670 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:15.739667 20713 consensus_queue.cc:237] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:15.733628 20700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s user 0.014s sys 0.021s
W20260317 16:18:15.703127 20709 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.008s sys 0.019s
W20260317 16:18:15.711222 20050 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.030s sys 0.047s
W20260317 16:18:15.681430 20616 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.012s sys 0.011s
W20260317 16:18:15.726890 20715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.804s user 0.022s sys 0.006s
I20260317 16:18:15.741070 20670 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.744500 20321 ts_tablet_manager.cc:1434] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 1.108s user 0.020s sys 0.012s
W20260317 16:18:15.734954 20236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.859s user 0.011s sys 0.061s
I20260317 16:18:15.759704 20700 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:15.723052 20536 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.008s sys 0.017s
I20260317 16:18:15.748900 20165 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:15.745695 20709 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.793s user 0.009s sys 0.020s
W20260317 16:18:15.737978 20120 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.012s sys 0.052s
W20260317 16:18:15.746352 20693 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.003s sys 0.019s
I20260317 16:18:15.757174 20428 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.758858 20485 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.762847 20320 ts_tablet_manager.cc:1434] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 1.098s user 0.045s sys 0.038s
W20260317 16:18:15.745441 20162 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.006s sys 0.019s
W20260317 16:18:15.756659 20321 ts_tablet_manager.cc:1467] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: Tablet startup took 1120ms
I20260317 16:18:15.760543 20700 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:15.788220 20320 ts_tablet_manager.cc:1467] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Tablet startup took 1123ms
W20260317 16:18:15.787722 20693 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.004s sys 0.024s
W20260317 16:18:15.756484 20715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.872s user 0.022s sys 0.007s
W20260317 16:18:15.792336 20016 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.008s sys 0.018s
W20260317 16:18:15.792750 20016 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.008s sys 0.018s
W20260317 16:18:15.755338 20616 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s user 0.012s sys 0.016s
W20260317 16:18:15.746655 20050 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.887s user 0.031s sys 0.048s
I20260317 16:18:15.788825 20740 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:15.792212 20162 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.006s sys 0.021s
W20260317 16:18:15.786870 20537 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.039s sys 0.020s
I20260317 16:18:15.772285 20709 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:15.765885 20670 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
W20260317 16:18:15.788782 20636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.003s sys 0.027s
W20260317 16:18:15.788547 20320 ts_tablet_manager.cc:1469] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Trace:
0317 16:18:04.255883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.256025 (+ 142us) service_pool.cc:225] Handling call
0317 16:18:04.259601 (+ 3576us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.259700 (+ 99us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.317830 (+ 58130us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:04.363979 (+ 46149us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:04.403910 (+ 39931us) inbound_call.cc:173] Queueing success response
0317 16:18:14.312324 (+9908414us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:14.664717 (+352393us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:14.819857 (+155140us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:14.834471 (+ 14614us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":23834,"raft.queue_time_us":391495,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":39992,"tablet-open.queue_time_us":9947944,"thread_start_us":857587,"threads_started":1}
I20260317 16:18:15.826020 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:15.826889 20164 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.009s sys 0.021s
I20260317 16:18:15.799737 20536 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.827997 20322 tablet_bootstrap.cc:492] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:15.837224 20322 ts_tablet_manager.cc:1403] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 1.427s user 0.047s sys 0.049s
W20260317 16:18:15.816643 20537 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.286s user 0.040s sys 0.020s
I20260317 16:18:15.791524 20428 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:15.802254 20628 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.798650 20485 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.801663 20674 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.843124 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:15.843258 20196 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:15.847496 20621 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:15.847599 20738 log.cc:927] Time spent T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20260317 16:18:15.848022 20164 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.010s sys 0.021s
I20260317 16:18:15.832736 20670 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:15.830833 20636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s user 0.004s sys 0.027s
W20260317 16:18:15.809139 20124 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.026s sys 0.025s
I20260317 16:18:15.853785 20536 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:15.791870 20321 ts_tablet_manager.cc:1469] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b: Trace:
0317 16:18:03.956715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:03.957650 (+ 935us) service_pool.cc:225] Handling call
0317 16:18:04.020982 (+ 63332us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e32ead1 00007fea3e30bd63 00007fea3e2ba4c0 00007fea37748bf4 00007fea37748b82 00007fea37748ab1
0317 16:18:04.106803 (+ 85821us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e32ead1 00007fea3e30bd63 00007fea3e2ba4c0 00007fea37748bf4 00007fea37748b82 00007fea37748ab1
0317 16:18:04.141770 (+ 34967us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.145002 (+ 3232us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.280858 (+135856us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:04.368782 (+ 87924us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:04.403093 (+ 34311us) inbound_call.cc:173] Queueing success response
0317 16:18:14.208788 (+9805695us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:14.636269 (+427481us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:14.885221 (+248952us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:14.886130 (+ 909us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":32095,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":40714368,"tablet-open.queue_time_us":9838860,"thread_start_us":822722,"threads_started":1}
I20260317 16:18:15.871344 20320 tablet_bootstrap.cc:492] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:15.836597 20616 consensus_queue.cc:1048] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260317 16:18:15.866706 20612 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.015s sys 0.015s
I20260317 16:18:15.865849 20748 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.912153 20699 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:15.876609 20124 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.825s user 0.027s sys 0.026s
I20260317 16:18:15.921046 20322 raft_consensus.cc:359] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:15.913372 20612 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.729s user 0.015s sys 0.015s
W20260317 16:18:15.922447 20195 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.007s sys 0.020s
I20260317 16:18:15.864041 20757 consensus_queue.cc:1048] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260317 16:18:15.919920 20699 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:15.923388 20699 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:15.920367 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:15.924525 20196 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
W20260317 16:18:15.925531 20195 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.008s sys 0.020s
I20260317 16:18:15.882876 20628 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.926192 20699 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.926517 20628 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:15.927199 20191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 69) took 1506 ms. Trace:
I20260317 16:18:15.882731 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:15.920729 20748 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:15.875226 20485 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:15.859834 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 66) took 1312 ms. Trace:
I20260317 16:18:15.918285 20700 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.868167 20734 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:15.865458 20745 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.922257 20322 raft_consensus.cc:385] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:15.878669 20751 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:15.941075 20485 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:15.943431 20322 raft_consensus.cc:740] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:15.944478 20321 tablet_bootstrap.cc:492] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:15.883008 20674 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:15.950801 20322 consensus_queue.cc:260] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:15.952023 20163 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.008s sys 0.028s
I20260317 16:18:15.942548 20734 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:15.982842 20734 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.976334 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:15.983945 20193 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:15.978001 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:15.984436 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:15.985368 20043 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
W20260317 16:18:15.985666 20163 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.967s user 0.009s sys 0.028s
I20260317 16:18:15.945964 20670 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:15.986956 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:15.998059 20734 consensus_queue.cc:1048] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:16.001466 20766 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:16.002262 20766 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:16.007519 20756 log.cc:927] Time spent T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
W20260317 16:18:16.009274 20758 log.cc:927] Time spent T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20260317 16:18:16.009536 20766 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.011507 20766 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.012247 20766 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.015949 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.020020 20628 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:15.955763 20536 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.022053 20041 raft_consensus.cc:2393] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:16.022254 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:15.985371 20048 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:16.044842 20192 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.022s sys 0.074s
I20260317 16:18:15.958175 20751 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.075088 20162 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:16.076838 20320 tablet_bootstrap.cc:654] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:16.074685 20192 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.049s user 0.023s sys 0.075s
I20260317 16:18:16.006282 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.074350 20194 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:15.936748 20050 rpcz_store.cc:276] 0317 16:18:14.538737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:14.539717 (+ 980us) service_pool.cc:225] Handling call
0317 16:18:14.571355 (+ 31638us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:14.571599 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:14.855450 (+283851us) log.cc:844] Serialized 14 byte log entry
0317 16:18:15.804598 (+949148us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:15.812310 (+ 7712us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:15.825200 (+ 12890us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:15.829958 (+ 4758us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:15.842137 (+ 12179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19883,"prepare.queue_time_us":530030,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":61640,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":1179612,"threads_started":2}
I20260317 16:18:16.102948 20764 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:16.108314 20764 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:16.108412 20015 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:16.108765 20764 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:16.110946 20755 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:16.111831 20428 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:15.957202 20674 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.104398 20322 ts_tablet_manager.cc:1434] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.266s user 0.011s sys 0.000s
I20260317 16:18:15.953969 20745 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.103150 20693 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.125319 20709 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:16.125455 20693 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:15.930750 20191 rpcz_store.cc:276] 0317 16:18:14.420396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:14.432978 (+ 12582us) service_pool.cc:225] Handling call
0317 16:18:14.439108 (+ 6130us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:14.608442 (+169334us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:14.610514 (+ 2072us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:14.646897 (+ 36383us) log.cc:844] Serialized 26 byte log entry
0317 16:18:15.395869 (+748972us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:15.395917 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:15.396001 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:15.926863 (+530862us) raft_consensus.cc:1685] finished
0317 16:18:15.926901 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:15.927084 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":69412,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":865245,"threads_started":2,"wal-append.queue_time_us":1011396}
I20260317 16:18:16.126900 20044 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:16.127224 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.127775 20693 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.128810 20322 tablet_bootstrap.cc:492] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:16.128995 20674 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.106928 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.131129 20195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 104) took 1311 ms. Trace:
I20260317 16:18:16.131552 20195 rpcz_store.cc:276] 0317 16:18:14.819628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:14.827371 (+ 7743us) service_pool.cc:225] Handling call
0317 16:18:14.859253 (+ 31882us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:15.223302 (+364049us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:15.223342 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:15.224070 (+ 728us) log.cc:844] Serialized 26 byte log entry
0317 16:18:15.926040 (+701970us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:15.926091 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:15.926177 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:16.115053 (+188876us) raft_consensus.cc:1685] finished
0317 16:18:16.123442 (+ 8389us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:16.131062 (+ 7620us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":364077,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":42132,"thread_start_us":1065147,"threads_started":2,"wal-append.queue_time_us":882669}
I20260317 16:18:16.134330 20746 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:16.105396 20751 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:16.135262 20746 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:16.135756 20746 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:16.105707 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.125999 20709 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.048012 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.076673 20621 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.093637 20730 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.092677 20197 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:16.114607 20692 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.106170 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.107717 20628 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:16.123991 20734 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:16.107621 20766 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.111599 20755 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:16.144428 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 352859us
I20260317 16:18:16.153169 20428 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.194540 20751 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:16.195385 20751 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:16.196305 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 452740us
I20260317 16:18:16.197999 20751 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.173051 20194 raft_consensus.cc:2393] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 2: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:16.201630 20238 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:16.153002 20199 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:16.256784 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.259994 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.265906 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.267086 20117 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:16.167245 20730 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:16.161827 20621 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.199998 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.204885 20751 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.171300 20692 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.207585 20536 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:16.202797 20238 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:16.206619 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.176524 20734 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.174706 20628 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:16.210038 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 444119us
I20260317 16:18:16.273190 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.273284 20693 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:16.201859 20428 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:16.305104 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 102419us
I20260317 16:18:16.276422 20748 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.273706 20049 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:16.286952 20536 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:16.315598 20318 tablet_bootstrap.cc:492] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:16.319643 20318 ts_tablet_manager.cc:1403] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 1.427s user 0.038s sys 0.041s
I20260317 16:18:16.299750 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.288379 20751 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.290807 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.326534 20322 tablet_bootstrap.cc:654] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:16.349663 20536 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:16.330796 20734 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:16.355357 20537 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:16.451808 20734 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:16.451922 20537 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.452337 20734 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:16.453977 20537 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.454073 20734 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.454210 20091 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:16.327245 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.316803 20043 raft_consensus.cc:2393] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:16.333812 20730 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.336625 20745 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.344085 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.313772 20700 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:16.470234 20090 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:16.359724 20120 raft_consensus.cc:1275] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:16.331459 20709 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.340045 20759 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:16.332156 20621 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.340911 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.365288 20748 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:16.335341 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.329072 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.348662 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.485949 20759 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.496510 20119 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:16.498091 20759 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.334167 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.498612 20264 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:16.498809 20194 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:16.499316 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.499636 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.497920 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.500830 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.501204 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.501828 20119 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:16.502044 20759 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:16.333602 20628 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:16.349433 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.502597 20759 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.352908 20777 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
W20260317 16:18:16.485307 20124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.009s sys 0.037s
W20260317 16:18:16.507220 20124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.009s sys 0.037s
I20260317 16:18:16.354372 20428 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.351725 20755 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.360150 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.346550 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.319180 20693 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:16.471509 20674 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:16.336319 20162 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:16.475258 20700 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:16.495918 20048 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:16.348152 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.499727 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.335704 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:16.482432 20744 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.022s sys 0.023s
I20260317 16:18:16.473711 20773 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.471280 20771 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:16.511207 20238 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:16.518054 20271 raft_consensus.cc:1275] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:16.475524 20745 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:16.480615 20692 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:16.482288 20537 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:16.570062 20700 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:16.589408 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.595288 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.596418 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.596670 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.597074 20195 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:16.597391 20536 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.598258 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.564406 20674 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.600268 20537 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.600706 20674 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:16.556185 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 403909us
I20260317 16:18:16.557155 20743 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:16.485332 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.508349 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.517724 20772 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:16.577040 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 65971.7us
I20260317 16:18:16.551047 20125 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
W20260317 16:18:16.613708 20764 consensus_meta.cc:287] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.504s user 0.015s sys 0.000s
W20260317 16:18:16.575788 20744 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.599s user 0.022s sys 0.024s
I20260317 16:18:16.614179 20764 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.476024 20670 consensus_queue.cc:1048] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260317 16:18:16.564296 20759 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.581810 20773 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.508414 20748 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:16.577446 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 241229us
I20260317 16:18:16.568166 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.579474 20194 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:16.507169 20777 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:16.580143 20628 raft_consensus.cc:2804] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:16.592581 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:16.553193 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 404661us
I20260317 16:18:16.594915 20318 raft_consensus.cc:359] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:16.584839 20745 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.610924 20751 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:16.604898 20536 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.612926 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.613432 20270 raft_consensus.cc:2393] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:16.614621 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.564817 20612 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:16.615343 20700 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.602614 20537 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.574322 20428 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.602713 20743 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:16.641652 20321 tablet_bootstrap.cc:654] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:16.585242 20771 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.603768 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:16.687541 20781 log.cc:927] Time spent T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260317 16:18:16.688208 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.582365 20050 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:16.659054 20318 raft_consensus.cc:385] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:16.612486 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.604099 20119 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:16.620855 20048 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:16.663290 20199 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:16.672768 20122 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:16.676040 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.604763 20772 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:16.730598 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 384886us
I20260317 16:18:16.699015 20771 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.684926 20428 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.711216 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.661412 20628 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:16.691458 20318 raft_consensus.cc:740] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:16.670432 20612 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:16.675273 20536 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:16.676167 20267 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:16.741076 20745 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.715718 20764 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.756723 20318 consensus_queue.cc:260] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:16.794132 20192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 94) took 1887 ms. Trace:
I20260317 16:18:16.743393 20269 raft_consensus.cc:2393] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:16.791406 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.727198 20121 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
W20260317 16:18:16.793987 20236 leader_election.cc:343] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:16.656845 20195 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
W20260317 16:18:16.748147 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 273452us
I20260317 16:18:16.794803 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.806268 20612 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:16.796346 20628 raft_consensus.cc:3060] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:16.803329 20536 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:16.830914 20320 tablet_bootstrap.cc:492] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:16.807269 20693 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:16.839094 20091 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:16.864099 20320 ts_tablet_manager.cc:1403] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 1.009s user 0.022s sys 0.052s
I20260317 16:18:16.865181 20318 ts_tablet_manager.cc:1434] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.545s user 0.006s sys 0.001s
W20260317 16:18:16.865585 20318 ts_tablet_manager.cc:1467] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Tablet startup took 545ms
W20260317 16:18:16.852739 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:16.816509 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.810883 20744 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:16.819950 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:16.855914 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 385723us
W20260317 16:18:16.856849 20117 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.012s sys 0.026s
I20260317 16:18:16.830829 20428 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.838177 20793 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:16.883433 20090 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:16.860888 20238 leader_election.cc:343] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:16.892987 20238 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:16.853997 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 60042.7us
I20260317 16:18:16.869617 20536 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:16.897001 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:16.899802 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:16.900369 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 110) took 2300 ms. Trace:
I20260317 16:18:16.901980 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.902889 20793 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.903262 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.904426 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.988782 20123 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:16.989176 20320 raft_consensus.cc:359] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.989801 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.905429 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:16.907439 20797 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:16.991319 20199 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:16.993418 20320 raft_consensus.cc:385] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:16.869876 20748 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.908107 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:16.883092 20117 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.012s sys 0.028s
I20260317 16:18:16.878918 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:16.866061 20318 ts_tablet_manager.cc:1469] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Trace:
0317 16:18:04.029818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.034141 (+ 4323us) service_pool.cc:225] Handling call
0317 16:18:04.089376 (+ 55235us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.089477 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.251598 (+162121us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:04.458445 (+206847us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:04.494744 (+ 36299us) inbound_call.cc:173] Queueing success response
0317 16:18:14.881554 (+10386810us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:16.320024 (+1438470us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:16.592900 (+272876us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:16.593433 (+ 533us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":5624,"raft.queue_time_us":13096,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":21633,"tablet-open.queue_time_us":10384519}
I20260317 16:18:16.869562 20590 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:16.881328 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.934000 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:17.021934 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 88033.3us
I20260317 16:18:17.022650 20016 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:16.934783 20238 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:16.882265 20771 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.883226 20744 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:16.932695 20773 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:16.895407 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:16.983937 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.984354 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:16.913010 20700 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:16.909047 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 67) took 1835 ms. Trace:
I20260317 16:18:17.006685 20777 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:16.950435 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 51802.6us
I20260317 16:18:16.980822 20795 consensus_queue.cc:1048] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
W20260317 16:18:16.987353 20794 log.cc:927] Time spent T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.133s user 0.003s sys 0.003s
I20260317 16:18:16.981009 20803 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:16.938009 20739 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:16.950939 20198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 100) took 1135 ms. Trace:
I20260317 16:18:16.988196 20701 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.082302 20701 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.082813 20701 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.062966 20124 rpcz_store.cc:276] 0317 16:18:15.048548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:15.048771 (+ 223us) service_pool.cc:225] Handling call
0317 16:18:15.049252 (+ 481us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:15.049493 (+ 241us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:15.921924 (+872431us) log.cc:844] Serialized 14 byte log entry
0317 16:18:16.823943 (+902019us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:16.830814 (+ 6871us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:16.832273 (+ 1459us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:16.868855 (+ 36582us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:16.884191 (+ 15336us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1062770,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":54,"raft.queue_time_us":93623,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":1770016,"threads_started":3,"wal-append.queue_time_us":682377}
I20260317 16:18:17.084899 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.085276 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:16.907656 20751 consensus_queue.cc:1048] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:17.087082 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.087270 20199 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:16.996058 20320 raft_consensus.cc:740] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:17.078510 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.088439 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.088850 20193 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
W20260317 16:18:17.009356 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 128663us
I20260317 16:18:17.089954 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.091471 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:17.021050 20746 consensus_meta.cc:287] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.881s user 0.015s sys 0.000s
I20260317 16:18:16.981133 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.048825 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:17.096858 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 48158.4us
W20260317 16:18:17.049409 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 114752us
I20260317 16:18:17.011545 20616 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:17.058012 20322 tablet_bootstrap.cc:492] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:17.089793 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.018648 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.042600 20789 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.043772 20811 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:17.048974 20692 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:16.982275 20088 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:17.056258 20748 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.087672 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.088680 20320 consensus_queue.cc:260] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:17.078981 20739 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:16.891781 20693 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:16.984086 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.024605 20590 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.086326 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.097935 20238 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:16.949566 20015 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:17.101347 20322 ts_tablet_manager.cc:1403] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.973s user 0.027s sys 0.007s
I20260317 16:18:17.070174 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.086529 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:17.137766 20757 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.125556 20739 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.141615 20757 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.119910 20119 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:17.092181 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.093014 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.105039 20771 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:17.134243 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.159389 20771 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.160960 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.161469 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.101059 20772 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:17.104483 20636 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:17.083086 20803 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.128557 20320 ts_tablet_manager.cc:1434] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.264s user 0.004s sys 0.004s
I20260317 16:18:17.131319 20238 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:17.122529 20811 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.121027 20709 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:17.132771 20197 raft_consensus.cc:2393] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:17.129503 20693 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:16.820427 20192 rpcz_store.cc:276] 0317 16:18:14.906303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:14.909030 (+ 2727us) service_pool.cc:225] Handling call
0317 16:18:14.987928 (+ 78898us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:16.115303 (+1127375us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:16.124384 (+ 9081us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:16.134334 (+ 9950us) log.cc:844] Serialized 26 byte log entry
0317 16:18:16.556904 (+422570us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:16.556975 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:16.557098 (+ 123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:16.793826 (+236728us) raft_consensus.cc:1685] finished
0317 16:18:16.793864 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:16.794059 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24378,"raft.queue_time_us":530377,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":11959,"thread_start_us":1451349,"threads_started":2,"wal-append.queue_time_us":656492}
W20260317 16:18:17.133111 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 150920us
I20260317 16:18:17.141217 20123 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:17.110852 20789 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.120824 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.082970 20744 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:17.065512 20046 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:17.121155 20692 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.123148 20748 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.100260 20746 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:17.118860 20047 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:17.132014 20121 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:17.110018 20239 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:17.165691 20238 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:17.248023 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 82362.3us
I20260317 16:18:17.203805 20789 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.153232 20757 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.228498 20692 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:17.228183 20748 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.190827 20709 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:17.154247 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 245855us
I20260317 16:18:17.136826 20809 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.162490 20116 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:17.164363 20636 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:17.261864 20048 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:17.261996 20636 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:17.082141 20198 rpcz_store.cc:276] 0317 16:18:15.809880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:15.813918 (+ 4038us) service_pool.cc:225] Handling call
0317 16:18:15.869967 (+ 56049us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:15.881520 (+ 11553us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:16.465996 (+584476us) log.cc:844] Serialized 14 byte log entry
0317 16:18:16.873697 (+407701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:16.886150 (+ 12453us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:16.894651 (+ 8501us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:16.910132 (+ 15481us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:16.944750 (+ 34618us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":664226,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":7635,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":9344,"thread_start_us":774096,"threads_started":2,"wal-append.queue_time_us":441941}
I20260317 16:18:17.264222 20321 tablet_bootstrap.cc:492] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:17.264756 20321 ts_tablet_manager.cc:1403] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.334s user 0.047s sys 0.099s
I20260317 16:18:17.267233 20803 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:17.267925 20321 raft_consensus.cc:359] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.268639 20321 raft_consensus.cc:385] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:17.268921 20321 raft_consensus.cc:740] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:17.269330 20746 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:17.269948 20746 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:17.207132 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 109677us
I20260317 16:18:17.205951 20322 raft_consensus.cc:359] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.180763 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.180600 20793 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:17.162657 20771 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:17.270452 20746 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.232493 20320 tablet_bootstrap.cc:492] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:17.271744 20322 raft_consensus.cc:385] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:17.154481 20701 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:17.204391 20125 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:17.226001 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.104498 20751 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:17.280355 20751 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.281271 20088 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:17.282042 20590 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:17.282346 20751 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:17.237282 20800 log.cc:927] Time spent T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.259s user 0.001s sys 0.001s
I20260317 16:18:17.282754 20088 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:17.283630 20751 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:17.284046 20751 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:17.284099 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.284505 20751 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:17.285359 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.271831 20746 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.287173 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.287961 20272 raft_consensus.cc:2393] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:17.269590 20321 consensus_queue.cc:260] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.181183 20772 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
W20260317 16:18:17.252574 20792 log.cc:927] Time spent T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.184s user 0.000s sys 0.001s
I20260317 16:18:17.288394 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 77) took 1088 ms. Trace:
I20260317 16:18:17.290796 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.272127 20322 raft_consensus.cc:740] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:16.944746 20042 rpcz_store.cc:276] 0317 16:18:14.592441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:14.668921 (+ 76480us) service_pool.cc:225] Handling call
0317 16:18:14.682813 (+ 13892us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:15.743098 (+1060285us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:15.745077 (+ 1979us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:15.791668 (+ 46591us) log.cc:844] Serialized 26 byte log entry
0317 16:18:16.247859 (+456191us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:16.827632 (+579773us) log.cc:844] Serialized 14 byte log entry
0317 16:18:16.857060 (+ 29428us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:16.868502 (+ 11442us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:16.876232 (+ 7730us) raft_consensus.cc:1685] finished
0317 16:18:16.879294 (+ 3062us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:16.893081 (+ 13787us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":338060,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":238227,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":9152896,"thread_start_us":1869155,"threads_started":3,"wal-append.queue_time_us":701234}
I20260317 16:18:17.146684 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:17.290911 20772 raft_consensus.cc:697] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:17.286528 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.253938 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.291893 20322 consensus_queue.cc:260] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.290678 20272 rpcz_store.cc:276] 0317 16:18:16.199868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.201936 (+ 2068us) service_pool.cc:225] Handling call
0317 16:18:17.287013 (+1085077us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:17.288314 (+ 1301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61282560}
I20260317 16:18:17.291740 20268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 83) took 1118 ms. Trace:
W20260317 16:18:17.292469 20775 log.cc:927] Time spent T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.789s user 0.003s sys 0.003s
I20260317 16:18:17.351665 20196 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.352069 20321 ts_tablet_manager.cc:1434] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.087s user 0.006s sys 0.002s
I20260317 16:18:17.260872 20809 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:17.273370 20777 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:17.277089 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:17.293387 20043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 72) took 1246 ms (client timeout 1638 ms). Trace:
I20260317 16:18:17.420331 20190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 99) took 1618 ms. Trace:
I20260317 16:18:17.420922 20190 rpcz_store.cc:276] 0317 16:18:15.801591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:15.803856 (+ 2265us) service_pool.cc:225] Handling call
0317 16:18:15.839199 (+ 35343us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:15.848121 (+ 8922us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:15.848152 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:15.848881 (+ 729us) log.cc:844] Serialized 26 byte log entry
0317 16:18:16.191123 (+342242us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:16.191196 (+ 73us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:16.191311 (+ 115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:17.291896 (+1100585us) spinlock_profiling.cc:243] Waited 9.05 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:17.419936 (+128040us) raft_consensus.cc:1685] finished
0317 16:18:17.419980 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:17.420259 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20840192,"thread_start_us":341702,"threads_started":1,"wal-append.queue_time_us":654429}
I20260317 16:18:17.276666 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:17.303694 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.423204 20050 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:17.427088 20267 raft_consensus.cc:2393] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:17.302690 20536 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:17.429199 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.429899 20536 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:17.294845 20803 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.360097 20268 rpcz_store.cc:276] 0317 16:18:16.204415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.210899 (+ 6484us) service_pool.cc:225] Handling call
0317 16:18:17.290607 (+1079708us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:17.291679 (+ 1072us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4562304}
W20260317 16:18:17.416615 20807 log.cc:927] Time spent T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.185s user 0.000s sys 0.000s
I20260317 16:18:17.416652 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.430150 20267 raft_consensus.cc:2368] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:17.299130 20766 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:17.408963 20777 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:17.278738 20117 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
W20260317 16:18:17.279075 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 180345us
I20260317 16:18:17.409109 20809 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.308089 20693 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:17.449581 20320 tablet_bootstrap.cc:654] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:17.455358 20322 ts_tablet_manager.cc:1434] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.321s user 0.017s sys 0.000s
W20260317 16:18:17.393671 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 68137.9us
I20260317 16:18:17.457229 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.284521 20088 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:17.446846 20814 log.cc:927] Time spent T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.309s user 0.000s sys 0.001s
I20260317 16:18:17.433275 20803 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:17.465703 20162 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.638s (SENT) [suppressed 3 similar messages]
I20260317 16:18:17.490314 20050 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:17.437417 20700 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:17.464485 20777 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:17.485170 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.487638 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.490024 20125 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:17.480063 20091 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:17.471997 20266 raft_consensus.cc:2368] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:17.433593 20015 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:17.510411 20803 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:17.420837 20043 rpcz_store.cc:269] 0317 16:18:16.046601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.048626 (+ 2025us) service_pool.cc:225] Handling call
0317 16:18:17.293184 (+1244558us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54001da454. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:17.293307 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250225536}
I20260317 16:18:17.495157 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.520364 20236 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:17.540367 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:17.540737 20236 leader_election.cc:343] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:17.541342 20236 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:17.541679 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.542111 20536 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:17.542729 20536 raft_consensus.cc:2749] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:17.542857 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:17.484163 20239 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:17.526018 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 46372.4us
I20260317 16:18:17.472370 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.457376 20772 consensus_queue.cc:237] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.472520 20766 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.504451 20190 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:17.515727 20771 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:17.525858 20322 tablet_bootstrap.cc:492] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:17.493468 20636 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:17.532197 20819 log.cc:927] Time spent T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.299s user 0.000s sys 0.002s
I20260317 16:18:17.550475 20795 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:17.531667 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 247257us
W20260317 16:18:17.578231 20165 leader_election.cc:343] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:17.535339 20196 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
W20260317 16:18:17.552594 20612 consensus_meta.cc:287] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.677s user 0.012s sys 0.001s
I20260317 16:18:17.518446 20700 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.528692 20811 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:17.607013 20766 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:17.607443 20743 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:17.606909 20748 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.608413 20636 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:17.609050 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.608616 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 74) took 1530 ms. Trace:
I20260317 16:18:17.612442 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:17.616530 20795 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.570330 20822 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:17.557404 20236 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:17.619195 20090 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:17.620260 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 141640us
W20260317 16:18:17.635408 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 151341us
W20260317 16:18:17.634378 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 56262.9us
I20260317 16:18:17.697731 20165 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:17.549211 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 141205us
I20260317 16:18:17.571457 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.709270 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.664754 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.711213 20125 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:17.711082 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:17.713022 20050 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:17.581496 20536 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:17.660068 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.649780 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:17.660343 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34240 (request call id 90) took 1330 ms. Trace:
I20260317 16:18:17.660867 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:17.715598 20264 rpcz_store.cc:276] 0317 16:18:16.329938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.500852 (+170914us) service_pool.cc:225] Handling call
0317 16:18:16.501169 (+ 317us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:16.674240 (+173071us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:16.674320 (+ 80us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:16.674383 (+ 63us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:16.674433 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:17.659712 (+985279us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b5400394254. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:17.659835 (+ 123us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:17.660227 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":393653504}
I20260317 16:18:17.716639 20190 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.717172 20826 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:17.717710 20826 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:17.664767 20046 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:17.666172 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.644171 20701 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:17.688057 20273 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:17.667865 20118 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 84) took 1380 ms. Trace:
I20260317 16:18:17.736210 20118 rpcz_store.cc:276] 0317 16:18:16.287441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.289869 (+ 2428us) service_pool.cc:225] Handling call
0317 16:18:17.132199 (+842330us) spinlock_profiling.cc:243] Waited 30.3 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:17.666983 (+534784us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:17.667187 (+ 204us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:17.667394 (+ 207us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:17.667426 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:17.667568 (+ 142us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:17.667604 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:17.667669 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:17.667806 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83437184}
I20260317 16:18:17.738351 20271 raft_consensus.cc:2368] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:17.667588 20789 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.740687 20789 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:17.669003 20318 tablet_bootstrap.cc:492] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:17.741292 20709 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:17.669451 20322 tablet_bootstrap.cc:654] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:17.686494 20827 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:17.747159 20827 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:17.669314 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:17.670058 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:17.671172 20238 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:17.673141 20816 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:17.531515 20265 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
W20260317 16:18:17.687065 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 68128.2us
I20260317 16:18:17.673491 20692 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:17.708459 20628 consensus_meta.cc:287] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.831s user 0.014s sys 0.000s
I20260317 16:18:17.715039 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.714388 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.684160 20795 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.646545 20612 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:17.684702 20120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 88) took 1226 ms. Trace:
W20260317 16:18:17.752353 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 81267.3us
I20260317 16:18:17.784016 20238 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:17.755229 20090 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:17.751368 20827 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:17.705175 20743 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:17.683213 20811 consensus_queue.cc:1059] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.070s
I20260317 16:18:17.676666 20746 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:17.754916 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 210375us
I20260317 16:18:17.771600 20048 raft_consensus.cc:2368] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
I20260317 16:18:17.742352 20709 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:17.751637 20796 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:17.765025 20739 consensus_meta.cc:287] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.609s user 0.004s sys 0.010s
I20260317 16:18:17.788079 20091 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:17.756965 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:17.788471 20238 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:17.789469 20047 raft_consensus.cc:1240] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:17.789934 20771 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:17.790176 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:17.790637 20751 consensus_meta.cc:287] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.001s
I20260317 16:18:17.792073 20318 tablet_bootstrap.cc:654] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:17.792793 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.793766 20826 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:17.761617 20766 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.796211 20612 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:17.797019 20122 raft_consensus.cc:2368] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3.
I20260317 16:18:17.798976 20239 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:17.800495 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.771929 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 113) took 1428 ms. Trace:
I20260317 16:18:17.802474 20321 tablet_bootstrap.cc:492] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:17.802580 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 83) took 1326 ms. Trace:
I20260317 16:18:17.766355 20701 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.804881 20090 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:17.765817 20192 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:17.812618 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:17.768649 20628 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:17.704819 20822 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.769930 20803 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.837561 19946 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46714 (request call id 38) took 4345 ms. Trace:
I20260317 16:18:17.793565 20748 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:17.772068 20050 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:17.760100 20757 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.829559 19945 catalog_manager.cc:5671] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:17.786093 20536 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
W20260317 16:18:17.853926 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 97093.1us
W20260317 16:18:17.854904 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 66951.3us
I20260317 16:18:17.804016 20692 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:17.876577 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 178964us
W20260317 16:18:17.828979 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 349237us
I20260317 16:18:17.856716 20670 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:17.928256 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 123525us
I20260317 16:18:17.877208 20766 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:17.876942 20751 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.930420 20796 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:17.860913 20793 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.829763 20743 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:17.876147 20816 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:17.841892 20709 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:17.857168 20739 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:17.656627 20045 rpcz_store.cc:276] 0317 16:18:16.077798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.094052 (+ 16254us) service_pool.cc:225] Handling call
0317 16:18:16.168900 (+ 74848us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:16.819964 (+651064us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:16.831751 (+ 11787us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:16.892172 (+ 60421us) log.cc:844] Serialized 26 byte log entry
0317 16:18:17.234164 (+341992us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:17.234230 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:17.592027 (+357797us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54000b3bd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:17.592099 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:17.592164 (+ 65us) raft_consensus.cc:1685] finished
0317 16:18:17.592187 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:17.608364 (+ 16177us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140516,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":285983616,"thread_start_us":606246,"threads_started":2,"wal-append.queue_time_us":292055}
W20260317 16:18:17.989893 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 201533us
W20260317 16:18:18.079324 20238 leader_election.cc:343] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:17.875850 20700 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:17.992879 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:17.935240 20771 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:17.928820 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 79320.6us
I20260317 16:18:18.045473 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.046664 20165 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:18.090807 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 45427.3us
W20260317 16:18:18.091105 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 44766.2us
I20260317 16:18:18.091528 20016 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.092171 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:18.051633 20751 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.052088 20796 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.052208 20320 tablet_bootstrap.cc:492] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:18.108374 20320 ts_tablet_manager.cc:1403] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.916s user 0.018s sys 0.026s
I20260317 16:18:18.052423 20793 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:18.044822 20766 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.053406 20267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 35) took 1120 ms. Trace:
W20260317 16:18:18.044978 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 173350us
W20260317 16:18:17.985731 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 276690us
I20260317 16:18:17.980265 20190 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:18.054396 20536 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:17.934671 20043 raft_consensus.cc:2393] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.044742 20748 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:17.989374 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.140543 20536 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.093883 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:18.096206 20047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 115) took 1752 ms. Trace:
W20260317 16:18:18.024333 20049 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 116) took 1650 ms (client timeout 1967 ms). Trace:
I20260317 16:18:18.060062 20816 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:18.060951 20041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 84) took 1800 ms (client timeout 1975 ms). Trace:
I20260317 16:18:17.782347 20120 rpcz_store.cc:276] 0317 16:18:16.453256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.504062 (+ 50806us) service_pool.cc:225] Handling call
0317 16:18:16.505406 (+ 1344us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:16.649608 (+144202us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:16.675825 (+ 26217us) log.cc:844] Serialized 14 byte log entry
0317 16:18:16.689443 (+ 13618us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:16.698116 (+ 8673us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:16.700614 (+ 2498us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:17.673952 (+973338us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54001f8bd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:17.674038 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:17.680007 (+ 5969us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28083,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":2991,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":8808,"spinlock_wait_cycles":238833408}
I20260317 16:18:18.083652 20124 raft_consensus.cc:2368] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
W20260317 16:18:17.985798 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 202319us
I20260317 16:18:17.886610 20046 rpcz_store.cc:276] 0317 16:18:16.343155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.257397 (+914242us) service_pool.cc:225] Handling call
0317 16:18:17.663933 (+406536us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:17.664345 (+ 412us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:17.749995 (+ 85650us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:17.769042 (+ 19047us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30812032}
I20260317 16:18:17.983552 19946 rpcz_store.cc:276] 0317 16:18:13.478997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:13.480820 (+ 1823us) service_pool.cc:225] Handling call
0317 16:18:17.824309 (+4343489us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:14.259086 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:14.275183 (+ 16097us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:14.275251 (+ 68us) write_op.cc:435] Acquired schema lock
0317 16:18:14.275278 (+ 27us) tablet.cc:661] Decoding operations
0317 16:18:14.291873 (+ 16595us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:14.291993 (+ 120us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:14.292027 (+ 34us) tablet.cc:684] Acquiring locks for 4 operations
0317 16:18:14.292527 (+ 500us) tablet.cc:700] Row locks acquired
0317 16:18:14.292550 (+ 23us) write_op.cc:260] PREPARE: finished
0317 16:18:14.292753 (+ 203us) write_op.cc:270] Start()
0317 16:18:14.292929 (+ 176us) write_op.cc:276] Timestamp: P: 1773764294292666 usec, L: 0
0317 16:18:14.292968 (+ 39us) op_driver.cc:348] REPLICATION: starting
0317 16:18:14.380439 (+ 87471us) log.cc:844] Serialized 1717 byte log entry
0317 16:18:15.847346 (+1466907us) op_driver.cc:464] REPLICATION: finished
0317 16:18:16.663114 (+815768us) write_op.cc:301] APPLY: starting
0317 16:18:16.663235 (+ 121us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:16.672615 (+ 9380us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:16.693000 (+ 20385us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:16.898380 (+205380us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:16.898710 (+ 330us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0317 16:18:16.898828 (+ 118us) write_op.cc:312] APPLY: finished
0317 16:18:16.899113 (+ 285us) log.cc:844] Serialized 40 byte log entry
0317 16:18:17.049124 (+150011us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:17.056100 (+ 6976us) write_op.cc:454] Released schema lock
0317 16:18:17.061773 (+ 5673us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:17.062310 (+ 537us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:17.062351 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":802790,"apply.run_cpu_time_us":45224,"apply.run_wall_time_us":404634,"num_ops":4,"prepare.queue_time_us":476114,"prepare.run_cpu_time_us":21896,"prepare.run_wall_time_us":363838,"raft.queue_time_us":971585,"raft.run_cpu_time_us":48590,"raft.run_wall_time_us":457321,"replication_time_us":1554192,"thread_start_us":1906816,"threads_started":5,"wal-append.queue_time_us":540267,"wal-append.run_cpu_time_us":39415,"wal-append.run_wall_time_us":2172488}]]}
I20260317 16:18:18.140439 20822 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:18.141317 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.093039 20016 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:18.144404 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 50613.6us
I20260317 16:18:17.986275 20777 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.119606 20827 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:18.142140 20048 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:18.131139 20269 raft_consensus.cc:2393] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:18.113512 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 89) took 1195 ms. Trace:
I20260317 16:18:18.114128 20701 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:17.927428 20270 rpcz_store.cc:276] 0317 16:18:16.467941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.500548 (+ 32607us) service_pool.cc:225] Handling call
0317 16:18:17.794579 (+1294031us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54006842d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:17.794736 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":244218112}
I20260317 16:18:18.115005 20826 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:18.115788 20793 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.151969 20321 tablet_bootstrap.cc:654] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:18.133317 20809 consensus_meta.cc:287] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.640s user 0.013s sys 0.001s
I20260317 16:18:18.208348 20811 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.208806 20809 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:18.115492 20734 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.119314 20692 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:18.127626 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.180362 20165 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:18.219314 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.144217 20042 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
W20260317 16:18:18.188513 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 95554.9us
I20260317 16:18:18.177162 20194 raft_consensus.cc:2393] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:18.193418 20320 raft_consensus.cc:359] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.207234 20701 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:18.207965 20832 log.cc:927] Time spent T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260317 16:18:18.214816 20091 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.178134 20816 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.186687 20827 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:18.192349 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.187088 20748 raft_consensus.cc:697] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:18.177585 20041 rpcz_store.cc:269] 0317 16:18:16.257291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.689266 (+431975us) service_pool.cc:225] Handling call
0317 16:18:17.812434 (+1123168us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:18.057949 (+245515us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10908416}
I20260317 16:18:18.236496 20088 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.211019 20826 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:18.268312 20826 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.268543 20612 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:18.268599 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.197791 20836 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:18.214716 20015 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.269623 20836 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:18.270085 20836 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:18.221966 20809 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:18.270458 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 71789.1us
I20260317 16:18:18.233517 20320 raft_consensus.cc:385] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:18.240145 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.273017 20320 raft_consensus.cc:740] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:18.236783 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.274206 20198 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.273967 20320 consensus_queue.cc:260] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.152727 20746 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:18.197481 20193 rpcz_store.cc:276] 0317 16:18:16.899281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.127659 (+228378us) service_pool.cc:225] Handling call
0317 16:18:17.179260 (+ 51601us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:17.226501 (+ 47241us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:17.518063 (+291562us) log.cc:844] Serialized 14 byte log entry
0317 16:18:18.023464 (+505401us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:18.047432 (+ 23968us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:18.057269 (+ 9837us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.073339 (+ 16070us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:18.094161 (+ 20822us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30630,"prepare.queue_time_us":329742,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":53478,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":13921,"thread_start_us":651356,"threads_started":3,"wal-append.queue_time_us":195580}
W20260317 16:18:18.150015 20049 rpcz_store.cc:269] 0317 16:18:16.344313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.792441 (+1448128us) service_pool.cc:225] Handling call
0317 16:18:17.994283 (+201842us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:18.276738 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.213201 20709 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:18.278694 20015 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:18.280755 20693 raft_consensus.cc:2749] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.280228 20709 consensus_queue.cc:1048] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:18.232086 20770 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:18.258417 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 43722.3us
I20260317 16:18:18.282423 20770 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.282920 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.283890 20193 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:18.261242 20803 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.269058 20612 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:18.270985 20822 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:18.198558 20838 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 16:18:18.232093 20834 log.cc:927] Time spent T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260317 16:18:18.237993 20016 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:18.289918 20238 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:18.220645 20692 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:18.220000 20272 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:18.232976 20116 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:18.151330 20047 rpcz_store.cc:276] 0317 16:18:16.344010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.257075 (+913065us) service_pool.cc:225] Handling call
0317 16:18:17.404158 (+147083us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:17.863073 (+458915us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.095780 (+232707us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b5400393d54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:18.096083 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":885591296}
I20260317 16:18:18.282923 20816 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.309603 20743 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:18.332176 20322 tablet_bootstrap.cc:492] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:18.332504 20816 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:18.332674 20322 ts_tablet_manager.cc:1403] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.816s user 0.038s sys 0.028s
W20260317 16:18:18.312187 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 74239.5us
I20260317 16:18:18.309533 20628 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:18.293876 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.237268 20840 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:18.290577 20751 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.288283 20827 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.330216 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.330528 20846 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
W20260317 16:18:18.336236 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 113613us
I20260317 16:18:18.269631 20041 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:18.320505 20692 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.313387 20822 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:18.238389 20190 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:18.326305 20746 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:18.365869 20840 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:18.387491 20318 tablet_bootstrap.cc:492] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:18.310639 20826 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:18.390681 20772 raft_consensus.cc:3055] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:18.368508 20266 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.391494 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 121) took 1555 ms. Trace:
I20260317 16:18:18.390524 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:18.403582 20844 log.cc:927] Time spent T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.271s user 0.001s sys 0.001s
I20260317 16:18:18.319965 20757 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:18.338256 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.294974 20612 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.327410 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:18.337260 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 75716.2us
I20260317 16:18:18.369665 20745 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:18.409433 20318 ts_tablet_manager.cc:1403] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.741s user 0.013s sys 0.026s
W20260317 16:18:18.410415 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 120637us
I20260317 16:18:18.414026 20042 rpcz_store.cc:276] 0317 16:18:16.839128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.431024 (+591896us) service_pool.cc:225] Handling call
0317 16:18:17.488251 (+ 57227us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:18.233620 (+745369us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.391225 (+157605us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54001b53d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:18.391431 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1454268288}
I20260317 16:18:18.338406 20739 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:18.411734 20320 ts_tablet_manager.cc:1434] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.303s user 0.017s sys 0.011s
I20260317 16:18:18.380328 20628 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:18.369402 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.395670 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.392007 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:18.400076 20795 consensus_meta.cc:287] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.615s user 0.010s sys 0.003s
I20260317 16:18:18.326146 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.388207 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 91) took 1098 ms. Trace:
I20260317 16:18:18.486658 20628 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.496191 20795 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.497556 20628 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.392060 20770 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:18.403545 20822 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.437493 20692 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:18.116129 20267 rpcz_store.cc:276] 0317 16:18:16.981090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.509229 (+528139us) service_pool.cc:225] Handling call
0317 16:18:17.537648 (+ 28419us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:17.558227 (+ 20579us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:17.789259 (+231032us) log.cc:844] Serialized 14 byte log entry
0317 16:18:18.044951 (+255692us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:18.053072 (+ 8121us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:18.053123 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.053208 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:18.053343 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19216,"prepare.queue_time_us":289470,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":78810,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":10629,"thread_start_us":325011,"threads_started":2,"wal-append.queue_time_us":249803}
I20260317 16:18:18.442102 20122 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 91) took 1481 ms. Trace:
I20260317 16:18:18.447304 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.408783 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:18.449467 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:18.412365 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.450812 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.451058 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.451949 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.453473 20709 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:18.451068 20194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34608 (request call id 88) took 1569 ms. Trace:
W20260317 16:18:18.455020 20842 log.cc:927] Time spent T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
I20260317 16:18:18.402313 20746 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:18.399115 20846 raft_consensus.cc:697] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:18.461627 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.419839 20162 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:18.463882 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:18.413312 20772 raft_consensus.cc:740] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:18.412997 20840 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:18.410732 20322 raft_consensus.cc:359] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.550053 20322 raft_consensus.cc:385] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:18.550377 20322 raft_consensus.cc:740] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:18.551071 20322 consensus_queue.cc:260] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.439867 20777 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:18.487346 20238 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:18.497320 20793 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:18.503963 20239 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:18.441021 20845 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:18.511143 20692 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:18.500689 20165 connection.cc:232] RPC call timeout handler was delayed by -1.1781089389999124s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
I20260317 16:18:18.568425 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.568981 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.569648 20197 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.569856 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.480104 20751 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:18.522967 20194 rpcz_store.cc:276] 0317 16:18:16.898870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.119685 (+220815us) service_pool.cc:225] Handling call
0317 16:18:17.789985 (+670300us) spinlock_profiling.cc:243] Waited 8.93 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:18.450824 (+660839us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54000b7554. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:18.450981 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":591707904}
I20260317 16:18:18.571035 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.571902 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.524844 20709 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.525338 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.526297 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.526929 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.527961 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:18.528674 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.532732 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.533349 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:18.534363 20835 log.cc:927] Time spent T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.398s user 0.004s sys 0.001s
I20260317 16:18:18.508663 20199 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:18.523567 20047 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.539561 20050 raft_consensus.cc:2393] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.395895 20748 consensus_queue.cc:237] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.542311 20746 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.572072 20322 ts_tablet_manager.cc:1434] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.239s user 0.006s sys 0.002s
I20260317 16:18:18.563621 20628 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:18.677207 20322 tablet_bootstrap.cc:492] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:18.678222 20709 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.679009 20265 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:18.680074 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.681222 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.682430 20322 tablet_bootstrap.cc:654] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:18.523828 20796 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.683434 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.683691 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:18.572569 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 85352.6us
W20260317 16:18:18.674713 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 109741us
I20260317 16:18:18.686743 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.682286 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:18.692023 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:18.692569 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.692865 20044 raft_consensus.cc:2393] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.693250 20119 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:18.694181 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.694645 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.695634 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.696482 20121 raft_consensus.cc:2393] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:18.692688 20041 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:18.694213 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.507424 20122 rpcz_store.cc:276] 0317 16:18:16.943072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.203782 (+260710us) service_pool.cc:225] Handling call
0317 16:18:17.666017 (+462235us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:18.412388 (+746371us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b540016d854. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:18.424373 (+ 11985us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112013440}
I20260317 16:18:18.557179 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.579491 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:18.552992 20118 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:18.572338 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.573217 20855 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:18.687676 20795 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:18.688445 20811 raft_consensus.cc:2804] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:18.714310 20855 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:18.714599 20811 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:18.714742 20855 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:18.715036 20811 raft_consensus.cc:3060] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:18.721143 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:18.705857 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 216766us
W20260317 16:18:18.684636 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 275809us
I20260317 16:18:18.708348 20734 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.700089 20190 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:18.706355 20826 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:18.731999 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.732295 20796 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.719667 20196 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
W20260317 16:18:18.739634 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 170900us
I20260317 16:18:18.711138 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.723376 20239 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:18.577718 20836 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:18.748191 20048 raft_consensus.cc:2393] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.748373 20123 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 104) took 1757 ms. Trace:
W20260317 16:18:18.729015 20236 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:18.767937 20016 leader_election.cc:400] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3.
I20260317 16:18:18.749562 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.750295 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:18.571035 20772 consensus_queue.cc:260] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:18.752342 20766 consensus_meta.cc:287] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.620s user 0.007s sys 0.003s
I20260317 16:18:18.752434 20197 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:18.725075 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 89) took 1459 ms. Trace:
I20260317 16:18:18.759552 20321 tablet_bootstrap.cc:492] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:18.787518 20321 ts_tablet_manager.cc:1403] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.985s user 0.046s sys 0.052s
I20260317 16:18:18.768332 20016 leader_election.cc:403] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:18.771276 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.758908 20162 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:18.768957 20117 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:18.775034 20236 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:18.752507 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:18.775929 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 52661.6us
I20260317 16:18:18.778829 20090 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.499374 20273 rpcz_store.cc:276] 0317 16:18:17.290234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.736255 (+446021us) service_pool.cc:225] Handling call
0317 16:18:17.736742 (+ 487us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:17.737052 (+ 310us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:17.994313 (+257261us) log.cc:844] Serialized 14 byte log entry
0317 16:18:18.320342 (+326029us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:18.325927 (+ 5585us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:18.339176 (+ 13249us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.374913 (+ 35737us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:18.385907 (+ 10994us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39380,"prepare.queue_time_us":220734,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":62640,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9610,"thread_start_us":425297,"threads_started":2,"wal-append.queue_time_us":211656}
I20260317 16:18:18.806182 20090 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:18.807057 20816 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.757691 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.808059 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.808370 20271 raft_consensus.cc:2393] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:18.809644 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:18.810446 20196 raft_consensus.cc:2393] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:18.787302 20796 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:18.760198 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.811957 20318 raft_consensus.cc:359] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.791208 20321 raft_consensus.cc:359] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.729427 20088 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.760054 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:18.793395 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 45966.9us
I20260317 16:18:18.837543 20016 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:18.788429 20165 leader_election.cc:343] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:18.838873 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 50494.2us
I20260317 16:18:18.840350 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:18.805299 20772 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:18.719333 20846 consensus_queue.cc:237] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:18.814338 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 71003.5us
I20260317 16:18:18.814906 20854 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:18.815371 20628 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:18.815811 20046 raft_consensus.cc:2393] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.819263 20321 raft_consensus.cc:385] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:18.768148 20701 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.819578 20318 raft_consensus.cc:385] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:18.880925 20318 raft_consensus.cc:740] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:18.822031 20193 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:18.822157 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 100) took 1930 ms. Trace:
I20260317 16:18:18.796238 20709 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:18.798368 20771 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:18.818651 20616 raft_consensus.cc:2749] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:18.788164 20766 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:18.796859 20827 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.574919 20845 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:18.847259 20165 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:18.817508 20796 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:18.856626 20016 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:18.938190 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 81661.8us
I20260317 16:18:18.855934 20803 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:18.878381 20321 raft_consensus.cc:740] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:18.944099 20321 consensus_queue.cc:260] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:18.941607 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 94598.4us
I20260317 16:18:18.889238 20047 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:18.807088 20836 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:18.843997 20793 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:18.882124 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:18.848827 20636 raft_consensus.cc:2804] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 16:18:18.835994 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 109525us
I20260317 16:18:18.853598 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.934937 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:18.940565 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:18.818460 20777 raft_consensus.cc:2749] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:18.878481 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:18.786499 20840 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:18.881479 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 168980us
W20260317 16:18:18.889634 20198 consensus_meta.cc:287] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.615s user 0.009s sys 0.004s
I20260317 16:18:18.884572 20628 raft_consensus.cc:697] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:18.975464 20267 raft_consensus.cc:2393] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:18.941236 20796 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:18.940465 20789 raft_consensus.cc:2749] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.984082 20191 raft_consensus.cc:2368] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
I20260317 16:18:18.998638 20822 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.001847 20049 raft_consensus.cc:2393] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:19.003502 20124 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 110) took 2010 ms (client timeout 2239 ms). Trace:
W20260317 16:18:19.004231 20124 rpcz_store.cc:269] 0317 16:18:16.992608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.229156 (+236548us) service_pool.cc:225] Handling call
0317 16:18:19.002940 (+1773784us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54001f86d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.003433 (+ 493us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1582413568}
I20260317 16:18:19.005383 20822 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:18.997689 20192 consensus_meta.cc:287] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 1.042s user 0.009s sys 0.006s
I20260317 16:18:19.006742 20236 leader_election.cc:400] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
W20260317 16:18:19.006676 20272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 40) took 2023 ms (client timeout 2638 ms). Trace:
I20260317 16:18:19.007660 20236 leader_election.cc:403] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:19.008085 20320 tablet_bootstrap.cc:492] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:19.008669 20236 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:19.005491 20238 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:18.966938 20612 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:18.892045 20266 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:19.011508 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.011900 20612 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:19.012192 20266 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:19.013228 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.014360 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:18.853366 20853 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:18.952351 20269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 36) took 1968 ms (client timeout 2025 ms). Trace:
I20260317 16:18:18.936825 20709 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:18.983300 20321 ts_tablet_manager.cc:1434] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.195s user 0.007s sys 0.001s
I20260317 16:18:18.958820 20636 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:19.006632 20192 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:18.936964 20811 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:19.041150 20192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55456 (request call id 127) took 1381 ms. Trace:
I20260317 16:18:19.041765 20192 rpcz_store.cc:276] 0317 16:18:17.659584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.669764 (+ 10180us) service_pool.cc:225] Handling call
0317 16:18:19.041079 (+1371315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:18.942975 20771 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:18.983830 20090 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:18.885042 20318 consensus_queue.cc:260] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:18.971401 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:18.966048 20268 raft_consensus.cc:1240] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260317 16:18:18.950996 20803 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:18.938881 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:19.049600 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 110846us
I20260317 16:18:19.050601 20016 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.887564 20116 rpcz_store.cc:276] 0317 16:18:16.892017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.306451 (+414434us) service_pool.cc:225] Handling call
0317 16:18:17.433986 (+127535us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:18.333856 (+899870us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.821800 (+487944us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54002e2554. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:18.822077 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1772504960}
I20260317 16:18:18.985663 20198 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:18.996951 20628 consensus_queue.cc:237] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:18.997056 20841 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:18.997563 20088 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:18.999821 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:18.952697 20838 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:19.067212 20239 leader_election.cc:400] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
W20260317 16:18:18.940618 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 49705.8us
I20260317 16:18:18.879527 20854 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:19.037261 20194 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
W20260317 16:18:19.033402 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 46505.7us
I20260317 16:18:19.035691 20091 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:18.988973 20119 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:19.082487 20670 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.061378 20836 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.015419 20193 raft_consensus.cc:1275] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:19.049309 20238 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:19.066617 20199 raft_consensus.cc:1240] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:19.042955 20757 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:19.062662 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 78891.6us
W20260317 16:18:19.039438 20269 rpcz_store.cc:269] 0317 16:18:16.981609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.303318 (+321709us) service_pool.cc:225] Handling call
0317 16:18:18.934531 (+1631213us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b5400418554. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:18.949792 (+ 15261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1474123008}
I20260317 16:18:19.100975 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.101161 20090 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:19.101773 20269 raft_consensus.cc:2368] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:19.102092 20816 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:19.102490 20816 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:19.102882 20816 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:19.102722 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.104074 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:19.106042 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.040954 20636 raft_consensus.cc:3060] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:19.035138 20853 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:19.037938 20536 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
I20260317 16:18:19.046279 20700 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:19.056145 20866 log.cc:927] Time spent T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260317 16:18:18.767599 20123 rpcz_store.cc:276] 0317 16:18:16.990972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.419752 (+428780us) service_pool.cc:225] Handling call
0317 16:18:17.672585 (+252833us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:17.672806 (+ 221us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:17.673196 (+ 390us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:17.673227 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:17.673809 (+ 582us) log.cc:844] Serialized 26 byte log entry
0317 16:18:17.957737 (+283928us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:18.468032 (+510295us) log.cc:844] Serialized 14 byte log entry
0317 16:18:18.480842 (+ 12810us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.487379 (+ 6537us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:18.533618 (+ 46239us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:18.742401 (+208783us) raft_consensus.cc:1685] finished
0317 16:18:18.744440 (+ 2039us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:18.748302 (+ 3862us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":494825,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":11914,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":20703,"spinlock_wait_cycles":82739200,"thread_start_us":609312,"threads_started":2,"wal-append.queue_time_us":382529}
I20260317 16:18:19.051363 20789 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.074914 20826 consensus_queue.cc:1048] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:18.787053 20264 rpcz_store.cc:276] 0317 16:18:17.260591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.717027 (+456436us) service_pool.cc:225] Handling call
0317 16:18:17.717358 (+ 331us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:17.717728 (+ 370us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:17.717762 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:17.718317 (+ 555us) log.cc:844] Serialized 26 byte log entry
0317 16:18:18.041550 (+323233us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:18.057378 (+ 15828us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:18.064720 (+ 7342us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:18.704388 (+639668us) raft_consensus.cc:1685] finished
0317 16:18:18.704409 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:18.719544 (+ 15135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9368,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":20418,"thread_start_us":273127,"threads_started":1,"wal-append.queue_time_us":366716}
I20260317 16:18:19.075264 20766 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:19.075941 20796 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.141163 20746 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.107743 20757 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:19.143780 20757 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:19.144830 20268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 87) took 1892 ms. Trace:
I20260317 16:18:19.076426 20771 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:18.994264 20869 consensus_queue.cc:1059] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260317 16:18:19.077864 20709 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:19.078130 20840 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:19.007782 20272 rpcz_store.cc:269] 0317 16:18:16.982970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.487313 (+504343us) service_pool.cc:225] Handling call
0317 16:18:19.006247 (+1518934us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b540028ebd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.006594 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1579728128}
I20260317 16:18:19.079030 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.032248 20125 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:19.052902 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.084088 20318 ts_tablet_manager.cc:1434] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.606s user 0.002s sys 0.006s
W20260317 16:18:19.159888 20318 ts_tablet_manager.cc:1467] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: Tablet startup took 681ms
W20260317 16:18:19.160279 20318 ts_tablet_manager.cc:1469] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f: Trace:
0317 16:18:04.378251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.378417 (+ 166us) service_pool.cc:225] Handling call
0317 16:18:04.382429 (+ 4012us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.383085 (+ 656us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.593378 (+210293us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:04.702515 (+109137us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:04.703198 (+ 683us) inbound_call.cc:173] Queueing success response
0317 16:18:17.668599 (+12965401us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:18.478450 (+809851us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:18.789560 (+311110us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:18.790234 (+ 674us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":5315,"raft.queue_time_us":13354,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":9868,"tablet-open.queue_time_us":12965457}
I20260317 16:18:19.085861 20701 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:19.086123 20854 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:19.087493 20321 tablet_bootstrap.cc:492] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:19.052663 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 44114.5us
I20260317 16:18:19.089394 20238 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:18.995247 20043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 130) took 1298 ms. Trace:
W20260317 16:18:19.090384 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 54778.1us
I20260317 16:18:19.067665 20612 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:19.173385 20612 raft_consensus.cc:697] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:19.174754 20612 consensus_queue.cc:237] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:19.109656 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.067478 20838 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:19.075974 20827 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.108088 20841 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:19.132877 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:19.099154 20745 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:19.091851 20670 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:19.134285 20873 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:19.145385 20268 rpcz_store.cc:276] 0317 16:18:17.251988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.558016 (+306028us) service_pool.cc:225] Handling call
0317 16:18:18.940298 (+1382282us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:19.050709 (+110411us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:19.144507 (+ 93798us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400418550. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.144753 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3129490816}
I20260317 16:18:19.115233 20700 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:19.163405 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 110935us
I20260317 16:18:19.166064 20320 tablet_bootstrap.cc:654] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:19.206393 20318 tablet_bootstrap.cc:492] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
W20260317 16:18:19.093991 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 96518.6us
I20260317 16:18:19.180158 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.092620 20165 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:19.210731 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 118216us
I20260317 16:18:19.211870 20165 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.115324 20853 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:19.142724 20796 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:19.183879 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 94591.9us
I20260317 16:18:19.148461 20845 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:19.135548 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 47101.4us
I20260317 16:18:19.147737 20041 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:19.165527 20811 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:19.167183 20789 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.203086 20322 tablet_bootstrap.cc:492] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:19.206844 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.188601 20846 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:19.213857 20125 raft_consensus.cc:2393] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:19.221794 20827 raft_consensus.cc:2804] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 16:18:19.243705 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 56041.5us
I20260317 16:18:19.191013 20838 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:19.210563 20700 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:19.233871 20322 ts_tablet_manager.cc:1403] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.557s user 0.015s sys 0.003s
I20260317 16:18:19.227274 20854 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:19.200110 20748 consensus_queue.cc:1059] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260317 16:18:19.213712 20745 raft_consensus.cc:697] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:19.231245 20693 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3.
I20260317 16:18:19.239207 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.182602 20043 rpcz_store.cc:276] 0317 16:18:17.695399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:17.710684 (+ 15285us) service_pool.cc:225] Handling call
0317 16:18:18.984421 (+1273737us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400729554. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:18.993408 (+ 8987us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331718528}
I20260317 16:18:19.246074 20239 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:19.207638 20841 raft_consensus.cc:697] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:19.228487 20236 leader_election.cc:343] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:19.248319 20827 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:19.214159 20670 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.240836 20692 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:19.274099 20692 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:19.275442 20670 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.276234 20692 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:19.247762 20846 raft_consensus.cc:697] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:19.220911 20165 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.247886 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.268277 20836 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.271925 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.241456 20822 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.235016 20616 consensus_queue.cc:1059] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260317 16:18:19.316568 20125 raft_consensus.cc:3060] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:19.317804 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 96941.9us
I20260317 16:18:19.318106 20854 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:19.278497 20827 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:19.278296 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:19.352403 20165 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.224023 20873 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:19.302637 20870 log.cc:927] Time spent T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260317 16:18:19.305115 20746 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.268666 20811 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
W20260317 16:18:19.300684 20044 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 77) took 3157 ms (client timeout 2062 ms). Trace:
I20260317 16:18:19.302912 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:19.300776 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.371194 20165 leader_election.cc:400] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:19.375550 20885 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:19.270417 20162 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:19.363219 20088 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:19.391170 20165 leader_election.cc:403] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:19.278088 20236 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:19.396245 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.371552 20854 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:19.383368 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:19.411723 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 94544.7us
I20260317 16:18:19.395948 20826 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:19.395152 20628 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:19.395156 20122 raft_consensus.cc:2468] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:19.367516 20745 consensus_queue.cc:237] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.409046 20692 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:19.413725 20771 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.408237 20840 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.395686 20268 raft_consensus.cc:2468] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
W20260317 16:18:19.387342 20044 rpcz_store.cc:269] 0317 16:18:16.143422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:16.147548 (+ 4126us) service_pool.cc:225] Handling call
0317 16:18:16.743640 (+596092us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:17.800339 (+1056699us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:19.277407 (+1477068us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b5400753d54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.300501 (+ 23094us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1857242496}
I20260317 16:18:19.416407 20709 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:19.409400 20757 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.388188 20841 consensus_queue.cc:237] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:19.432874 20887 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:19.433358 20878 log.cc:927] Time spent T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.157s user 0.002s sys 0.000s
W20260317 16:18:19.421161 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 159071us
I20260317 16:18:19.430822 20125 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:19.422094 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:19.444306 20239 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:19.420928 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 91038.8us
I20260317 16:18:19.423599 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:19.422603 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 196486us
I20260317 16:18:19.443542 20709 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
W20260317 16:18:19.452368 20886 log.cc:927] Time spent T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260317 16:18:19.461362 20322 raft_consensus.cc:359] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:19.489323 20322 raft_consensus.cc:385] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:19.449460 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:19.466995 20883 log.cc:927] Time spent T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260317 16:18:19.474330 20873 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.496999 20091 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:19.498718 20016 leader_election.cc:343] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:19.479328 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.482969 20321 tablet_bootstrap.cc:654] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:19.415241 20846 consensus_queue.cc:237] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.485080 20239 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:19.448014 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:19.435006 20692 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.430794 20628 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.492002 20318 tablet_bootstrap.cc:654] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:19.463968 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:19.489991 20322 raft_consensus.cc:740] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:19.460017 20811 raft_consensus.cc:2749] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:19.478920 20887 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.499097 20885 raft_consensus.cc:697] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:19.500033 20238 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:19.489326 20709 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:19.488729 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:19.525786 20322 consensus_queue.cc:260] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:19.495054 20757 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:19.478852 20700 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:19.544212 20090 leader_election.cc:343] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:19.506795 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:19.506822 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:19.507136 20612 raft_consensus.cc:2804] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:19.553437 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.554075 20125 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:19.540740 20855 consensus_meta.cc:287] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.825s user 0.011s sys 0.004s
I20260317 16:18:19.518066 20088 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:19.528548 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:19.493024 20636 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:19.548193 20122 raft_consensus.cc:2368] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:19.560109 20320 tablet_bootstrap.cc:492] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:19.572172 20320 ts_tablet_manager.cc:1403] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.564s user 0.029s sys 0.010s
W20260317 16:18:19.554461 20772 consensus_meta.cc:287] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.699s user 0.012s sys 0.001s
I20260317 16:18:19.551327 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.556456 20612 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:19.594802 20892 consensus_queue.cc:1059] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:19.595038 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:19.596937 20636 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:19.598285 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:19.599051 20748 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.604986 20090 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:19.603540 20165 leader_election.cc:336] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Timed out: RequestConsensusVote RPC to 127.16.123.196:43213 timed out after 2.026s (SENT)
I20260317 16:18:19.615742 20125 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:19.608623 20165 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:19.654527 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 49690.6us
I20260317 16:18:19.574198 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:19.571817 20811 raft_consensus.cc:2804] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:19.572407 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:19.602474 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.668378 20826 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:19.661181 20616 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.576009 20891 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.604104 20612 raft_consensus.cc:3060] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:19.573208 20757 raft_consensus.cc:2804] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:19.608412 20117 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:19.654336 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 113509us
I20260317 16:18:19.599440 20885 consensus_queue.cc:237] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.561463 20855 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:19.683712 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 75781.4us
W20260317 16:18:19.667595 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 64109.5us
I20260317 16:18:19.603904 20816 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.662142 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:19.674576 20811 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:19.688486 20045 raft_consensus.cc:1240] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:19.673986 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:19.679082 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 102392us
I20260317 16:18:19.705801 20320 raft_consensus.cc:359] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:19.712291 20320 raft_consensus.cc:385] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:19.712800 20320 raft_consensus.cc:740] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:19.697104 20165 leader_election.cc:400] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:19.713954 20165 leader_election.cc:403] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260317 16:18:19.689096 20123 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:19.698109 20757 raft_consensus.cc:697] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:19.716315 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:19.678145 20893 consensus_queue.cc:1059] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260317 16:18:19.709614 20811 raft_consensus.cc:3060] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:19.718451 20320 consensus_queue.cc:260] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:19.688122 20162 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:19.726717 20265 raft_consensus.cc:2393] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:19.726524 20840 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:19.734933 20840 raft_consensus.cc:697] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:19.735925 20840 consensus_queue.cc:237] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:19.717974 20826 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:19.732560 20268 raft_consensus.cc:2393] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:19.730563 20122 raft_consensus.cc:2393] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:19.721082 20692 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.731892 20628 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:19.728223 20757 consensus_queue.cc:237] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:19.760011 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 260328us
W20260317 16:18:19.760953 20238 leader_election.cc:343] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:19.737829 20700 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:19.763684 20320 ts_tablet_manager.cc:1434] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.191s user 0.010s sys 0.000s
I20260317 16:18:19.752396 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.744657 20873 raft_consensus.cc:2749] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:19.850059 20322 ts_tablet_manager.cc:1434] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.602s user 0.019s sys 0.019s
I20260317 16:18:19.810374 20321 tablet_bootstrap.cc:492] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:19.763798 20238 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:20.010816 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 247100us
W20260317 16:18:20.008231 20322 ts_tablet_manager.cc:1467] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: Tablet startup took 760ms
I20260317 16:18:20.014410 20321 ts_tablet_manager.cc:1403] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.958s user 0.014s sys 0.030s
W20260317 16:18:19.933735 20853 consensus_meta.cc:287] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.706s user 0.010s sys 0.000s
I20260317 16:18:19.850972 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:19.857674 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.028754 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:19.932753 20709 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.031625 20192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 97) took 1298 ms. Trace:
I20260317 16:18:20.031723 20042 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.032330 20198 raft_consensus.cc:1240] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.033036 20199 raft_consensus.cc:1275] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260317 16:18:19.916787 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 153805us
I20260317 16:18:19.811151 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:19.928771 20612 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.035458 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:19.830577 20838 consensus_meta.cc:287] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.567s user 0.000s sys 0.012s
I20260317 16:18:20.035658 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:19.759579 20746 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:20.036888 20612 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.929062 20117 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:20.037410 20746 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.038021 20116 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.038054 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:19.762145 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 82153.6us
I20260317 16:18:19.930675 20271 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:20.040324 20162 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.034340 20709 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.042147 20162 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:19.930137 20827 consensus_meta.cc:287] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.558s user 0.012s sys 0.001s
W20260317 16:18:19.821118 20902 log.cc:927] Time spent T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260317 16:18:20.029778 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:19.855172 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.043182 20827 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:19.856231 20194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 115) took 1452 ms. Trace:
I20260317 16:18:20.044147 20195 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.044381 20194 rpcz_store.cc:276] 0317 16:18:18.403819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.571539 (+167720us) service_pool.cc:225] Handling call
0317 16:18:19.856004 (+1284465us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400505d54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.856152 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788106624}
I20260317 16:18:19.930919 20041 raft_consensus.cc:1275] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.045578 19945 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46692 (request call id 38) took 6065 ms. Trace:
I20260317 16:18:19.821272 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 90) took 2119 ms. Trace:
I20260317 16:18:20.039666 20746 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:19.922820 20871 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:19.956780 20816 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:19.972491 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:19.967000 20803 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.072101 20266 raft_consensus.cc:2393] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
W20260317 16:18:20.011545 20322 ts_tablet_manager.cc:1469] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321: Trace:
0317 16:18:04.866036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.866361 (+ 325us) service_pool.cc:225] Handling call
0317 16:18:04.868519 (+ 2158us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.868631 (+ 112us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.928992 (+ 60361us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.099736 (+170744us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.100620 (+ 884us) inbound_call.cc:173] Queueing success response
0317 16:18:18.676866 (+13576246us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:19.248234 (+571368us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:19.452941 (+204707us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:19.453840 (+ 899us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":8697,"raft.queue_time_us":114856,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":18070,"tablet-open.queue_time_us":13576292,"thread_start_us":180033,"threads_started":1}
W20260317 16:18:20.029285 20881 log.cc:927] Time spent T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.638s user 0.006s sys 0.004s
W20260317 16:18:20.079356 20120 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 113) took 1250 ms (client timeout 1616 ms). Trace:
W20260317 16:18:19.856835 20267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 98) took 1376 ms (client timeout 1656 ms). Trace:
I20260317 16:18:20.030567 20121 raft_consensus.cc:1240] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.036960 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:19.917631 20855 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:20.086035 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 362046us
I20260317 16:18:20.025753 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.036266 20048 raft_consensus.cc:2393] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:20.087970 20120 rpcz_store.cc:269] 0317 16:18:18.828518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.834865 (+ 6347us) service_pool.cc:225] Handling call
0317 16:18:19.854953 (+1020088us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:20.079260 (+224307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10858240}
I20260317 16:18:20.032125 20192 rpcz_store.cc:276] 0317 16:18:18.732907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:19.043243 (+310336us) service_pool.cc:225] Handling call
0317 16:18:19.043597 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:19.091784 (+ 48187us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:19.107806 (+ 16022us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:19.109462 (+ 1656us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:19.115434 (+ 5972us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.031367 (+915933us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b54001c75d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:20.031415 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:20.031567 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91649920}
I20260317 16:18:20.097692 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.097924 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.098585 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 111) took 1270 ms. Trace:
I20260317 16:18:20.099581 20124 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:20.091631 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.100661 20117 raft_consensus.cc:2393] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:20.101616 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.102860 20840 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:20.103371 20840 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.106204 20840 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.099594 20047 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:20.086159 20871 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.107818 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.108978 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.109154 20871 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.072566 20046 rpcz_store.cc:276] 0317 16:18:17.701657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.192655 (+490998us) service_pool.cc:225] Handling call
0317 16:18:19.814242 (+1621587us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b540061bf54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.821176 (+ 6934us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1178739968}
I20260317 16:18:20.073078 20873 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.102916 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.110502 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.111263 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.111928 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.112224 20318 tablet_bootstrap.cc:492] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:20.112434 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.112715 20318 ts_tablet_manager.cc:1403] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.914s user 0.019s sys 0.006s
I20260317 16:18:20.113487 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.113736 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:20.030205 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 1.27425e+06us
I20260317 16:18:20.111299 20046 raft_consensus.cc:2393] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:20.112715 20043 raft_consensus.cc:2393] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:20.116355 20043 raft_consensus.cc:1275] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.116969 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 129) took 2421 ms. Trace:
I20260317 16:18:20.117267 20046 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.117411 20045 rpcz_store.cc:276] 0317 16:18:17.695082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.208146 (+513064us) service_pool.cc:225] Handling call
0317 16:18:19.301035 (+1092889us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:19.713461 (+412426us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.115974 (+402513us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b5400753d54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:20.116774 (+ 800us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400753d50. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.116910 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3323761408}
I20260317 16:18:20.119292 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 109) took 1243 ms. Trace:
I20260317 16:18:20.123211 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 99) took 1638 ms. Trace:
I20260317 16:18:20.091264 20044 raft_consensus.cc:3060] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:20.088250 20320 tablet_bootstrap.cc:492] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:20.025581 20826 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:20.092537 20016 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.090708 20321 raft_consensus.cc:359] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.137300 20321 raft_consensus.cc:385] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:20.040382 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 45) took 1205 ms. Trace:
I20260317 16:18:20.071961 20050 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:19.854532 20190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 122) took 1077 ms. Trace:
W20260317 16:18:20.086249 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 44199.7us
I20260317 16:18:20.098692 20048 raft_consensus.cc:2393] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:20.148919 20123 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.109747 20047 raft_consensus.cc:2393] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:20.149863 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:20.150177 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.151243 20236 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.151749 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 44) took 1317 ms. Trace:
I20260317 16:18:20.119968 20318 raft_consensus.cc:359] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.154511 20318 raft_consensus.cc:385] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:20.045456 20827 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.072328 20628 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.152918 20855 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.156929 20845 raft_consensus.cc:3055] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.157362 20845 raft_consensus.cc:740] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.159178 20748 raft_consensus.cc:3055] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:20.159610 20748 raft_consensus.cc:740] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.160380 20748 consensus_queue.cc:260] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:19.933823 20811 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.124197 20264 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.123697 20265 rpcz_store.cc:276] 0317 16:18:18.485064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.682042 (+196978us) service_pool.cc:225] Handling call
0317 16:18:20.122957 (+1440915us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400775bd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.123137 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2354784000}
I20260317 16:18:20.114693 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.119683 20116 rpcz_store.cc:276] 0317 16:18:18.875638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:19.066089 (+190451us) service_pool.cc:225] Handling call
0317 16:18:19.708049 (+641960us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:20.037816 (+329767us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:20.038527 (+ 711us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.119105 (+ 80578us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b540004f1d0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.119236 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756863104}
I20260317 16:18:20.161671 20748 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.132177 20239 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:20.017761 20700 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.036598 20853 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.137596 20321 raft_consensus.cc:740] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:20.184542 20321 consensus_queue.cc:260] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.036453 20838 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.118331 20015 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:20.148473 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.099025 20121 rpcz_store.cc:276] 0317 16:18:18.827773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.848504 (+ 20731us) service_pool.cc:225] Handling call
0317 16:18:20.030053 (+1181549us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:20.030271 (+ 218us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:20.095044 (+ 64773us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.098525 (+ 3481us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8857344}
I20260317 16:18:20.193044 20693 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:20.193485 20693 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:20.193903 20693 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.193504 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.154847 20318 raft_consensus.cc:740] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:20.152176 20271 rpcz_store.cc:276] 0317 16:18:18.834413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.843844 (+ 9431us) service_pool.cc:225] Handling call
0317 16:18:20.151573 (+1307729us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b5400684cd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.151688 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1457513728}
I20260317 16:18:20.198268 20016 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.198331 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.155592 20892 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.127499 20897 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:20.156679 20855 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:20.181314 20777 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.206007 20777 consensus_queue.cc:1059] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260317 16:18:20.186199 20191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55456 (request call id 133) took 1688 ms (client timeout 2231 ms). Trace:
I20260317 16:18:20.073118 19945 rpcz_store.cc:276] 0317 16:18:13.980702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:13.980943 (+ 241us) service_pool.cc:225] Handling call
0317 16:18:20.045504 (+6064561us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:18.476225 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:18.685238 (+209013us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:18.687966 (+ 2728us) write_op.cc:435] Acquired schema lock
0317 16:18:18.690891 (+ 2925us) tablet.cc:661] Decoding operations
0317 16:18:18.728444 (+ 37553us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:18.728518 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:18.728566 (+ 48us) tablet.cc:684] Acquiring locks for 1 operations
0317 16:18:18.731602 (+ 3036us) tablet.cc:700] Row locks acquired
0317 16:18:18.731622 (+ 20us) write_op.cc:260] PREPARE: finished
0317 16:18:18.731768 (+ 146us) write_op.cc:270] Start()
0317 16:18:18.731904 (+ 136us) write_op.cc:276] Timestamp: P: 1773764298731722 usec, L: 0
0317 16:18:18.731924 (+ 20us) op_driver.cc:348] REPLICATION: starting
0317 16:18:18.732697 (+ 773us) log.cc:844] Serialized 543 byte log entry
0317 16:18:19.085861 (+353164us) op_driver.cc:464] REPLICATION: finished
0317 16:18:19.189898 (+104037us) write_op.cc:301] APPLY: starting
0317 16:18:19.190011 (+ 113us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:19.190135 (+ 124us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:19.190292 (+ 157us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:19.219390 (+ 29098us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:19.219581 (+ 191us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 16:18:19.219637 (+ 56us) write_op.cc:312] APPLY: finished
0317 16:18:19.261393 (+ 41756us) log.cc:844] Serialized 22 byte log entry
0317 16:18:19.271833 (+ 10440us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:19.284138 (+ 12305us) write_op.cc:454] Released schema lock
0317 16:18:19.301020 (+ 16882us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:19.375321 (+ 74301us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:19.375365 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":142711,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":103720,"apply.run_cpu_time_us":14308,"apply.run_wall_time_us":185373,"num_ops":1,"prepare.queue_time_us":365426,"prepare.run_cpu_time_us":63655,"prepare.run_wall_time_us":412405,"raft.queue_time_us":80698,"raft.run_cpu_time_us":49366,"raft.run_wall_time_us":70695,"replication_time_us":353772,"thread_start_us":500962,"threads_started":4,"wal-append.queue_time_us":214881}]]}
I20260317 16:18:20.175460 20751 consensus_queue.cc:1059] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260317 16:18:20.208160 20897 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:20.208448 20191 rpcz_store.cc:269] 0317 16:18:18.496037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.572066 (+ 76029us) service_pool.cc:225] Handling call
0317 16:18:20.175806 (+1603740us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54003f4dd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.184396 (+ 8590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558744448}
I20260317 16:18:20.167029 20165 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:20.090832 20267 rpcz_store.cc:269] 0317 16:18:18.480669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.515968 (+ 35299us) service_pool.cc:225] Handling call
0317 16:18:19.856578 (+1340610us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400419454. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:19.856763 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":585456896}
W20260317 16:18:20.169656 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:20.131682 20898 consensus_queue.cc:1048] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260317 16:18:20.139544 20746 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:20.201087 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 52) took 1208 ms. Trace:
I20260317 16:18:20.239252 20264 rpcz_store.cc:276] 0317 16:18:18.992274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:19.159815 (+167541us) service_pool.cc:225] Handling call
0317 16:18:20.123501 (+963686us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:20.123979 (+ 478us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:20.179046 (+ 55067us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.198591 (+ 19545us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10256128}
I20260317 16:18:20.145939 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:20.178267 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 46175.8us
I20260317 16:18:20.163460 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.188954 20822 consensus_queue.cc:1059] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:20.158242 20845 consensus_queue.cc:260] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.197175 20318 consensus_queue.cc:260] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.200995 20320 tablet_bootstrap.cc:654] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:20.208060 20700 raft_consensus.cc:3055] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.208645 20269 raft_consensus.cc:2393] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:20.167479 20196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 93) took 1715 ms. Trace:
I20260317 16:18:20.177656 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.211205 20321 ts_tablet_manager.cc:1434] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.196s user 0.009s sys 0.000s
I20260317 16:18:20.211642 20165 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:20.266098 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 54567.2us
I20260317 16:18:20.268381 20165 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.270260 20165 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.110003 20195 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:20.156703 20816 consensus_queue.cc:1048] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260317 16:18:20.149057 20088 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.215022 20272 raft_consensus.cc:1240] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.147819 20190 rpcz_store.cc:276] 0317 16:18:18.777503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:19.045291 (+267788us) service_pool.cc:225] Handling call
0317 16:18:19.045622 (+ 331us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:19.072283 (+ 26661us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:19.086495 (+ 14212us) log.cc:844] Serialized 14 byte log entry
0317 16:18:19.215157 (+128662us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:19.220513 (+ 5356us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:19.227746 (+ 7233us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:19.734489 (+506743us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b18000c7900. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36252383 00007fea36254751 00007fea36254709 00007fea36254699 00007fea36254661 00007fea3625395d 0000000000379254 0000000000379099 00007fea3623574a
0317 16:18:19.853098 (+118609us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b18000c7900. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36220386 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9
0317 16:18:19.854127 (+ 1029us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b54003f61d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:19.854206 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:19.854418 (+ 212us) inbound_call.cc:173] Queueing success response
0317 16:18:20.028757 (+174339us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
Metrics: {"prepare.queue_time_us":4909,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":8872,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":5994,"spinlock_wait_cycles":87398656,"thread_start_us":107484,"threads_started":1,"wal-append.queue_time_us":243564}
I20260317 16:18:20.216269 20238 leader_election.cc:400] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Vote denied by peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885) with higher term. Message: Invalid argument: T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3.
I20260317 16:18:20.216777 20855 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.217253 20193 raft_consensus.cc:1275] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260317 16:18:20.288969 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 140017us
I20260317 16:18:20.290215 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:20.291702 20612 consensus_queue.cc:1048] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:20.292119 20887 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.292203 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:20.293309 20193 raft_consensus.cc:2393] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:20.293502 20770 raft_consensus.cc:3055] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.293941 20770 raft_consensus.cc:740] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:20.218583 20811 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:20.294751 20770 consensus_queue.cc:260] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.295778 20770 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.296556 20195 raft_consensus.cc:1240] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.219010 20322 tablet_bootstrap.cc:492] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:20.220693 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.221122 20853 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.217792 20117 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.184038 20162 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:20.197836 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 79620.3us
I20260317 16:18:20.307325 20015 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.197978 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.309366 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.254801 20892 consensus_queue.cc:1059] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260317 16:18:20.310041 20266 raft_consensus.cc:2376] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2. Re-sending same reply.
I20260317 16:18:20.222151 20044 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:20.267679 20628 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.222952 20891 raft_consensus.cc:3055] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:20.311340 20891 raft_consensus.cc:740] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:20.312129 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 135770us
I20260317 16:18:20.224210 20042 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:20.225003 20536 raft_consensus.cc:3055] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
W20260317 16:18:20.322820 20909 log.cc:927] Time spent T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.116s user 0.000s sys 0.002s
I20260317 16:18:20.322907 20840 raft_consensus.cc:2749] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.322817 20536 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.324190 20811 consensus_queue.cc:1059] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:20.264715 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.326756 20119 raft_consensus.cc:2393] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:20.228052 20265 raft_consensus.cc:1240] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260317 16:18:20.328186 20822 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.138540 20273 rpcz_store.cc:276] 0317 16:18:18.834786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.843772 (+ 8986us) service_pool.cc:225] Handling call
0317 16:18:19.850521 (+1006749us) spinlock_profiling.cc:243] Waited 50.1 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:20.040313 (+189792us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115123072}
I20260317 16:18:20.312242 20891 consensus_queue.cc:260] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.212277 20692 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.330672 20891 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.230875 20124 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:20.212901 20827 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.216796 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.273450 20090 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.271710 20616 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.274801 20871 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:20.275460 20318 ts_tablet_manager.cc:1434] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.162s user 0.007s sys 0.000s
I20260317 16:18:20.275842 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.268018 20197 raft_consensus.cc:2393] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
W20260317 16:18:20.276553 20118 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 115) took 1431 ms (client timeout 1651 ms). Trace:
I20260317 16:18:20.276968 20873 consensus_queue.cc:1059] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:20.257165 20700 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.263067 20777 consensus_queue.cc:1059] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
W20260317 16:18:20.287811 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 71693.2us
I20260317 16:18:20.321645 20193 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.290683 20771 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.185621 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.310461 20892 raft_consensus.cc:3055] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.353854 20816 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.257565 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.323118 20046 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.233886 20897 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.326138 20536 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:20.348680 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 75317.4us
I20260317 16:18:20.273874 20826 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.356206 20321 tablet_bootstrap.cc:492] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:20.261138 20196 rpcz_store.cc:276] 0317 16:18:18.452548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.822114 (+369566us) service_pool.cc:225] Handling call
0317 16:18:19.003511 (+181397us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:19.003745 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:19.003960 (+ 215us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:19.004048 (+ 88us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:19.004093 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:19.004181 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:20.167266 (+1163085us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540005b9d0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.167400 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":416264064}
I20260317 16:18:20.364152 20238 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:20.319567 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 59633.1us
I20260317 16:18:20.341197 20612 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:20.223322 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:20.352991 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 110) took 1473 ms. Trace:
I20260317 16:18:20.353627 20871 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:20.358525 20264 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:20.354152 20616 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.376317 20268 raft_consensus.cc:2410] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260317 16:18:20.389573 20318 tablet_bootstrap.cc:492] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:20.322891 20162 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.325044 20693 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.378697 20892 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.367336 20088 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:20.389963 20270 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 102) took 1353 ms (client timeout 1500 ms). Trace:
W20260317 16:18:20.353001 20118 rpcz_store.cc:269] 0317 16:18:18.845569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.848557 (+ 2988us) service_pool.cc:225] Handling call
0317 16:18:20.141319 (+1292762us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:20.276460 (+135141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8739712}
I20260317 16:18:20.415181 20822 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.419934 20120 raft_consensus.cc:2393] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 2: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:20.460407 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 137637us
I20260317 16:18:20.462785 20892 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.403626 20612 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.495558 20892 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.463430 20838 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.403028 20700 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.465085 20536 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:20.520053 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 157261us
I20260317 16:18:20.520534 20318 tablet_bootstrap.cc:654] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:20.468538 20693 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:20.469376 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 116508us
I20260317 16:18:20.485256 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:20.473374 20238 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:20.405267 20887 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:20.405449 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:20.483711 20827 raft_consensus.cc:3055] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.493549 20616 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.408974 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:20.516363 20770 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:20.400985 20124 rpcz_store.cc:276] 0317 16:18:18.876932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.025261 (+1148329us) service_pool.cc:225] Handling call
0317 16:18:20.350603 (+325342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:20.462785 20873 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.402921 20871 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:20.530542 20322 tablet_bootstrap.cc:654] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:20.420989 20236 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.496061 20162 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:20.260175 20846 consensus_queue.cc:1175] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "795c49e9595f45bdb8d6b568498abf1f" responder_term: 3 status { last_received { term: 2 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260317 16:18:20.572481 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:20.470347 20270 rpcz_store.cc:269] 0317 16:18:19.035192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:19.057201 (+ 22009us) service_pool.cc:225] Handling call
0317 16:18:20.216613 (+1159412us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:20.388365 (+171752us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35685248}
I20260317 16:18:20.505736 20822 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.592098 20320 tablet_bootstrap.cc:492] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:20.598629 20320 ts_tablet_manager.cc:1403] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.514s user 0.029s sys 0.014s
I20260317 16:18:20.546130 20827 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:20.551755 20887 raft_consensus.cc:697] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.574422 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:20.599970 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 78539.6us
I20260317 16:18:20.609341 20016 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:20.584259 20770 raft_consensus.cc:697] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:20.601917 20273 raft_consensus.cc:2393] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:20.596485 20855 raft_consensus.cc:2749] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260317 16:18:20.613924 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 73388.8us
I20260317 16:18:20.616688 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.600191 20612 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:20.621021 20693 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:20.651284 20693 raft_consensus.cc:697] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:20.652343 20693 consensus_queue.cc:237] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:20.596387 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 187389us
I20260317 16:18:20.655632 20855 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.655611 20236 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.656323 20855 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:20.614813 20267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 108) took 1116 ms. Trace:
W20260317 16:18:20.628803 20912 log.cc:927] Time spent T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260317 16:18:20.605437 20873 raft_consensus.cc:2804] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:20.635867 20922 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:20.639984 20915 log.cc:927] Time spent T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
W20260317 16:18:20.643169 20913 log.cc:927] Time spent T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
W20260317 16:18:20.612591 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 116595us
W20260317 16:18:20.647552 20088 leader_election.cc:343] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.649367 20197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 113) took 2252 ms. Trace:
I20260317 16:18:20.649762 20822 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:20.650499 20266 raft_consensus.cc:2393] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:20.625305 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.656163 20827 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:20.670862 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 62) took 1246 ms. Trace:
I20260317 16:18:20.663815 20536 raft_consensus.cc:2749] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.681200 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.682221 20049 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:20.662570 20922 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:20.666841 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 358738us
I20260317 16:18:20.703685 20270 raft_consensus.cc:1240] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.704522 20322 tablet_bootstrap.cc:492] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:20.673105 20197 rpcz_store.cc:276] 0317 16:18:18.397193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.812868 (+415675us) service_pool.cc:225] Handling call
0317 16:18:20.649138 (+1836270us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400506754. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:20.649297 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2954839936}
I20260317 16:18:20.705742 20320 raft_consensus.cc:359] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.671849 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.707197 20041 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:20.708537 20628 consensus_queue.cc:1059] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:20.708957 20898 raft_consensus.cc:3055] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.709466 20898 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.709852 20628 raft_consensus.cc:3055] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:20.710258 20628 raft_consensus.cc:740] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:20.668012 20873 raft_consensus.cc:697] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:20.711081 20628 consensus_queue.cc:260] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.712492 20628 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.681097 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:20.714309 20920 log.cc:927] Time spent T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260317 16:18:20.714949 20838 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:20.683328 20271 rpcz_store.cc:276] 0317 16:18:19.424107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.242943 (+818836us) service_pool.cc:225] Handling call
0317 16:18:20.243509 (+ 566us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:20.244603 (+ 1094us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:20.244638 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:20.245183 (+ 545us) log.cc:844] Serialized 26 byte log entry
0317 16:18:20.272513 (+ 27330us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:20.286791 (+ 14278us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.303729 (+ 16938us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:20.628054 (+324325us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:20.670498 (+ 42444us) raft_consensus.cc:1685] finished
0317 16:18:20.670531 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:20.670799 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2226,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":87,"spinlock_wait_cycles":23580928,"thread_start_us":17037,"threads_started":1,"wal-append.queue_time_us":162081}
I20260317 16:18:20.715523 20838 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.716254 20827 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.716995 20041 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:20.718016 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34240 (request call id 110) took 1436 ms. Trace:
I20260317 16:18:20.711630 20873 consensus_queue.cc:237] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.721033 20049 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.718429 20272 rpcz_store.cc:276] 0317 16:18:19.329261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.192082 (+862821us) service_pool.cc:225] Handling call
0317 16:18:20.192557 (+ 475us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:20.286280 (+ 93723us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.717719 (+431439us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400418f54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:20.717955 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51102080}
I20260317 16:18:20.658030 20267 rpcz_store.cc:276] 0317 16:18:19.496332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.216739 (+720407us) service_pool.cc:225] Handling call
0317 16:18:20.233707 (+ 16968us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:20.233943 (+ 236us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:20.409097 (+175154us) log.cc:844] Serialized 14 byte log entry
0317 16:18:20.584691 (+175594us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:20.595169 (+ 10478us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:20.596957 (+ 1788us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:20.602678 (+ 5721us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:20.612599 (+ 9921us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":863,"prepare.queue_time_us":133928,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":9430,"raft.queue_time_us":4554,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":10136,"thread_start_us":285753,"threads_started":2,"wal-append.queue_time_us":204396}
I20260317 16:18:20.723608 20838 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.693897 20770 consensus_queue.cc:237] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.705384 20322 ts_tablet_manager.cc:1403] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.487s user 0.009s sys 0.010s
I20260317 16:18:20.688017 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.716248 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.624400 20887 consensus_queue.cc:237] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:20.680087 20162 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:20.730048 20836 consensus_queue.cc:1059] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260317 16:18:20.693413 20855 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.737422 20836 raft_consensus.cc:3055] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:20.738036 20836 raft_consensus.cc:740] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:20.738852 20836 consensus_queue.cc:260] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:20.739861 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:20.740154 20836 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.740762 20322 raft_consensus.cc:359] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.753268 20322 raft_consensus.cc:385] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:20.732287 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:20.732688 20811 raft_consensus.cc:3055] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:20.733475 20692 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:20.733713 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.696502 20922 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:20.742936 20869 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:20.694422 20884 log.cc:927] Time spent T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 1.307s user 0.005s sys 0.010s
W20260317 16:18:20.749022 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:20.738624 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:20.749779 20919 log.cc:927] Time spent T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260317 16:18:20.726070 20771 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.780898 20117 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.751688 20822 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.706476 20320 raft_consensus.cc:385] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:20.781934 20822 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:20.784101 20822 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:20.784839 20049 raft_consensus.cc:2393] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:20.785598 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:20.788321 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.789220 20318 tablet_bootstrap.cc:492] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:20.789739 20318 ts_tablet_manager.cc:1403] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.405s user 0.020s sys 0.005s
I20260317 16:18:20.762897 20321 tablet_bootstrap.cc:654] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:20.768877 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.799307 20267 raft_consensus.cc:1275] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:20.770192 20045 raft_consensus.cc:1275] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.777462 20091 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.776474 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.779129 20050 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:20.801234 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.775882 20869 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.801895 20042 raft_consensus.cc:1275] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:20.802881 20845 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.803583 20869 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:20.780655 20118 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.782346 20320 raft_consensus.cc:740] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:20.737402 20162 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.753928 20322 raft_consensus.cc:740] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:20.812955 20322 consensus_queue.cc:260] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.767659 20015 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.795256 20047 raft_consensus.cc:1240] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.817502 20320 consensus_queue.cc:260] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:20.822247 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 44827.4us
W20260317 16:18:20.824978 20091 leader_election.cc:343] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.806886 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:20.808096 20165 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.825989 20091 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.770398 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.822116 20771 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.710408 20898 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.832491 20121 raft_consensus.cc:1275] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.764029 20811 raft_consensus.cc:740] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:20.833464 20700 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:20.835050 20897 consensus_queue.cc:1175] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9a3afb098d6a45f0b405566771a1157b" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260317 16:18:20.835733 20090 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.822552 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:20.801743 20045 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.804483 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.834259 20811 consensus_queue.cc:260] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.839694 20124 raft_consensus.cc:2410] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:20.840045 20811 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.840548 20922 consensus_queue.cc:1048] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:20.828240 20236 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:20.842610 20236 leader_election.cc:343] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.843041 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.843293 20748 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:20.843351 20236 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.843809 20269 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:20.843962 20748 consensus_queue.cc:1048] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:20.844426 20869 consensus_queue.cc:1048] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:20.845091 20272 raft_consensus.cc:1240] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.828948 20746 raft_consensus.cc:3055] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.827186 20199 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:20.846402 20746 raft_consensus.cc:740] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:20.843410 20922 raft_consensus.cc:2749] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:20.847332 20746 consensus_queue.cc:260] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.848292 20746 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.848451 20091 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.848901 20320 ts_tablet_manager.cc:1434] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.250s user 0.009s sys 0.001s
I20260317 16:18:20.849344 20771 raft_consensus.cc:2749] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:20.849596 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 112306us
I20260317 16:18:20.850081 20320 tablet_bootstrap.cc:492] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:20.849969 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.850903 20162 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:20.817654 20046 raft_consensus.cc:1275] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.852860 20162 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:20.850020 20771 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.830339 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.831377 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:20.762122 20826 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.848552 20123 raft_consensus.cc:1275] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.829370 20322 ts_tablet_manager.cc:1434] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.102s user 0.006s sys 0.001s
I20260317 16:18:20.861603 20049 raft_consensus.cc:1275] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.818361 20853 consensus_queue.cc:1048] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260317 16:18:20.854313 20838 raft_consensus.cc:2804] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:20.811455 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:20.832553 20898 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.873114 20116 raft_consensus.cc:2393] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:20.874133 20853 raft_consensus.cc:2804] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 16:18:20.859848 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:20.874642 20853 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:20.875094 20853 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:20.879470 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.880090 20838 raft_consensus.cc:697] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:20.880291 20125 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:20.881258 20318 raft_consensus.cc:359] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:20.860633 20236 leader_election.cc:343] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.893007 20318 raft_consensus.cc:385] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:20.893406 20318 raft_consensus.cc:740] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:20.853713 20840 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:20.825125 19948 catalog_manager.cc:5671] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 2, leader changed from d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196) to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:20.866195 20239 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:20.798499 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.871025 20322 tablet_bootstrap.cc:492] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:20.894225 20318 consensus_queue.cc:260] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.881083 20838 consensus_queue.cc:237] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:20.882031 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 120893us
I20260317 16:18:20.900666 20269 raft_consensus.cc:1275] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.901954 20015 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.883257 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.899247 20893 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.894397 20897 consensus_queue.cc:1048] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:20.903903 20925 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.904480 20015 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.904546 20925 consensus_queue.cc:1059] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:20.854499 20120 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.905170 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:20.855435 20122 raft_consensus.cc:1240] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:20.899813 20840 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:20.907080 20840 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:20.907492 20917 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:20.884696 20628 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:20.909880 20628 raft_consensus.cc:697] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:20.910324 20198 raft_consensus.cc:1275] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.864316 20887 consensus_queue.cc:1048] T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260317 16:18:20.885771 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:20.911695 20320 tablet_bootstrap.cc:654] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:20.912108 20272 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:20.887522 20826 consensus_queue.cc:1048] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:20.888258 20194 raft_consensus.cc:1240] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.864832 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:20.888608 20264 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:20.819380 20871 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:20.860064 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:20.864355 20822 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260317 16:18:20.917450 20239 leader_election.cc:343] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.923889 20091 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:20.914983 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 46001.8us
W20260317 16:18:20.928721 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 120723us
I20260317 16:18:20.925073 20692 consensus_queue.cc:1059] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260317 16:18:20.902383 20869 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260317 16:18:20.923427 20628 consensus_queue.cc:237] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:20.932116 20871 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:20.932523 20195 raft_consensus.cc:1240] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:20.932927 20612 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:20.934558 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 74575.4us
I20260317 16:18:20.936092 20318 ts_tablet_manager.cc:1434] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.146s user 0.009s sys 0.000s
I20260317 16:18:20.933787 20693 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.939884 20042 raft_consensus.cc:1275] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.937286 20822 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:20.933286 20924 consensus_queue.cc:1048] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260317 16:18:20.953586 20612 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:20.944783 20194 raft_consensus.cc:1275] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:20.948256 20195 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:20.936041 20700 consensus_queue.cc:1059] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
W20260317 16:18:20.949882 20238 leader_election.cc:343] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.968947 20238 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.951210 20015 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:20.951570 20917 raft_consensus.cc:3055] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:20.944280 20836 raft_consensus.cc:3055] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:20.976380 20836 raft_consensus.cc:740] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:20.936103 19948 catalog_manager.cc:5671] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
W20260317 16:18:20.979103 20238 leader_election.cc:343] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.979097 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.979841 20238 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:20.965826 20016 leader_election.cc:343] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.975593 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:20.972193 20917 raft_consensus.cc:740] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:20.958396 20822 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:20.981076 20238 leader_election.cc:343] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.979306 19948 catalog_manager.cc:5671] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 2, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:20.966748 20612 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:20.995388 20238 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.989576 20165 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.993623 20897 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:20.995026 20318 tablet_bootstrap.cc:492] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
W20260317 16:18:20.997028 20238 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.997594 20238 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.996202 19948 catalog_manager.cc:5671] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 2, leader changed from 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195) to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 2 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } }
W20260317 16:18:21.003921 20165 leader_election.cc:343] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.957366 20693 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:20.983099 20887 raft_consensus.cc:2749] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:21.006294 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:21.006125 19948 catalog_manager.cc:5671] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:21.018298 20165 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:21.018591 19948 catalog_manager.cc:5671] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 2, leader changed from 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195) to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 2 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } }
I20260317 16:18:21.019491 20165 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:20.988117 20826 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.987468 20751 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:20.977335 20836 consensus_queue.cc:260] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:21.023142 20090 leader_election.cc:343] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:20.996356 20822 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:20.988143 20869 consensus_queue.cc:1048] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260317 16:18:21.021204 20873 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:21.000588 20700 raft_consensus.cc:2749] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:20.972868 20853 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:21.000995 20893 raft_consensus.cc:2749] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:21.027132 20090 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:21.032130 20318 tablet_bootstrap.cc:654] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:21.020865 20827 raft_consensus.cc:2749] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:21.006321 20935 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:21.023159 20916 raft_consensus.cc:2749] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260317 16:18:21.099886 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 75221.1us
I20260317 16:18:21.003401 20748 consensus_queue.cc:1048] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260317 16:18:21.003154 20757 consensus_queue.cc:1059] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260317 16:18:21.070418 20770 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:21.087883 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:21.022164 20628 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260317 16:18:21.043331 20751 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:21.092836 20322 tablet_bootstrap.cc:654] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:21.094583 20321 tablet_bootstrap.cc:492] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:21.127241 20321 ts_tablet_manager.cc:1403] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.779s user 0.053s sys 0.007s
I20260317 16:18:21.089051 20853 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:21.118714 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 137567us
I20260317 16:18:21.026037 20937 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:21.031591 20936 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:21.120162 20891 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.065600 20897 consensus_queue.cc:1059] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260317 16:18:21.034096 20826 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.115437 20869 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.066341 20871 consensus_queue.cc:1059] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260317 16:18:21.124514 20893 raft_consensus.cc:3055] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:21.219992 20893 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:21.220422 20871 raft_consensus.cc:3055] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:21.109391 20892 consensus_queue.cc:1048] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260317 16:18:21.198468 20826 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:21.191076 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:21.199563 20321 raft_consensus.cc:359] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.244915 20321 raft_consensus.cc:385] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:21.200695 20917 consensus_queue.cc:260] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:21.201006 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:21.202425 20939 log.cc:927] Time spent T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260317 16:18:21.203166 20845 raft_consensus.cc:3055] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:21.204226 20891 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:21.129946 20771 raft_consensus.cc:2749] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:21.189889 20897 consensus_queue.cc:1243] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer 7a8150828b9c4db7b4cdcf2de9f35096 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260317 16:18:21.209028 20935 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:21.193307 20933 log.cc:927] Time spent T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.222s user 0.008s sys 0.000s
I20260317 16:18:21.210000 20693 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:21.214157 20751 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:21.217511 20612 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.218015 20320 tablet_bootstrap.cc:492] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:21.219174 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:21.072784 20119 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
W20260317 16:18:21.191872 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 60039.6us
I20260317 16:18:21.220878 20893 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.133464 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:21.232798 20194 raft_consensus.cc:1240] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:21.245405 20321 raft_consensus.cc:740] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:21.235082 20871 raft_consensus.cc:740] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
W20260317 16:18:21.261180 20165 leader_election.cc:343] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:21.266258 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:21.250036 20692 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.271697 20320 ts_tablet_manager.cc:1403] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.422s user 0.012s sys 0.006s
I20260317 16:18:21.234798 20869 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:21.262988 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:21.255283 20845 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:21.278615 20236 leader_election.cc:343] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:21.286867 20322 tablet_bootstrap.cc:492] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:21.287243 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:21.289106 20321 consensus_queue.cc:260] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.294209 20236 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:21.270957 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 319822us
I20260317 16:18:21.296145 20016 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:21.274853 20917 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:21.297222 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:21.295739 20322 ts_tablet_manager.cc:1403] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.425s user 0.024s sys 0.008s
I20260317 16:18:21.267493 20935 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.298949 20898 consensus_queue.cc:1048] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260317 16:18:21.302867 20050 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:21.305472 20869 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:21.306689 20871 consensus_queue.cc:260] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.308491 20612 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:21.320746 20836 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:21.316164 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:21.302486 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:21.333242 20925 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:21.305521 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:21.299602 20897 raft_consensus.cc:2749] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:21.312459 20891 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:21.291054 20692 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:21.291220 20195 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:21.313978 20321 ts_tablet_manager.cc:1434] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.186s user 0.008s sys 0.001s
I20260317 16:18:21.301405 20893 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:21.355410 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:21.309235 20845 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.313351 20123 raft_consensus.cc:3060] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:21.299408 20693 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.316491 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:21.314026 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:21.317863 20015 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:21.384945 20320 raft_consensus.cc:359] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.386227 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:21.388389 20692 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:21.389479 20321 tablet_bootstrap.cc:492] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:21.389807 20840 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:21.390424 20949 log.cc:927] Time spent T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260317 16:18:21.391820 20090 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:21.474186 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 82487.1us
W20260317 16:18:21.434952 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 100888us
I20260317 16:18:21.379132 20048 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:21.469103 20320 raft_consensus.cc:385] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:21.476859 20320 raft_consensus.cc:740] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:21.488252 20320 consensus_queue.cc:260] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:21.492309 20236 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:21.496534 20050 raft_consensus.cc:1275] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:21.498678 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:21.442824 20898 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.471577 20042 raft_consensus.cc:2393] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:21.508754 20322 raft_consensus.cc:359] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.509996 20322 raft_consensus.cc:385] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:21.510380 20322 raft_consensus.cc:740] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:21.511677 20322 consensus_queue.cc:260] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:21.508349 20239 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:21.513581 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:21.514431 20239 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:21.515208 20125 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
W20260317 16:18:21.488085 20953 log.cc:927] Time spent T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260317 16:18:21.487574 20891 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:21.519354 20162 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:21.521378 20826 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:21.521907 20826 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:21.522254 20826 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:21.475399 20266 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:21.480062 20268 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:21.470680 20693 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.565555 20318 tablet_bootstrap.cc:492] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:21.546082 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:21.572304 20318 ts_tablet_manager.cc:1403] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.589s user 0.017s sys 0.028s
I20260317 16:18:21.581127 20322 ts_tablet_manager.cc:1434] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.278s user 0.009s sys 0.000s
I20260317 16:18:21.582217 20322 tablet_bootstrap.cc:492] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:21.584996 20692 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:21.585395 20318 raft_consensus.cc:359] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.586257 20318 raft_consensus.cc:385] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:21.587559 20091 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:21.586592 20318 raft_consensus.cc:740] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:21.523518 20916 raft_consensus.cc:2749] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:21.516680 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 196390us
I20260317 16:18:21.589816 20320 ts_tablet_manager.cc:1434] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.307s user 0.005s sys 0.003s
I20260317 16:18:21.529737 20198 raft_consensus.cc:1240] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:21.519551 20891 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:21.594934 20771 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:21.595067 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:21.595579 20771 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:21.596046 20771 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:21.596145 20891 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:21.543023 20757 raft_consensus.cc:2749] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:21.602864 20194 raft_consensus.cc:1275] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:21.603948 20922 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.604508 20922 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:21.607184 20320 tablet_bootstrap.cc:492] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:21.607147 20757 consensus_queue.cc:1048] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:21.608369 20840 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:21.601275 20628 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.610704 20628 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.612218 20048 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:21.613093 20628 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:21.613255 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:21.513527 20898 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:21.616305 20125 raft_consensus.cc:1275] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260317 16:18:21.620353 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:21.593968 20961 log.cc:927] Time spent T 9a97b610497d410fb77dc8d4e5aca57e P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260317 16:18:21.591135 20318 consensus_queue.cc:260] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:21.638948 20048 raft_consensus.cc:1275] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:21.646225 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:21.599277 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:21.627982 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 118735us
I20260317 16:18:21.668463 20318 ts_tablet_manager.cc:1434] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.096s user 0.007s sys 0.000s
I20260317 16:18:21.668432 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:21.590972 20692 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:21.640863 20916 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:21.649448 20935 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:21.664717 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:21.628912 20628 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260317 16:18:21.680135 20125 raft_consensus.cc:1275] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:21.652345 20116 raft_consensus.cc:2468] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
W20260317 16:18:21.699362 20016 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:21.700888 20318 tablet_bootstrap.cc:492] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:21.674998 20916 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:21.671070 20264 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:21.703248 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:21.697881 20268 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:21.732792 20320 tablet_bootstrap.cc:654] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:21.735783 20321 tablet_bootstrap.cc:654] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:21.702312 20925 raft_consensus.cc:697] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:21.670948 20162 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:21.768882 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:21.726258 20916 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:21.771001 20125 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:21.730468 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:21.731796 20898 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:21.800583 20091 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:21.801779 20967 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260317 16:18:21.789918 20965 log.cc:927] Time spent T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20260317 16:18:21.729231 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 62911.5us
I20260317 16:18:21.744851 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:21.806015 20692 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:21.902279 20322 tablet_bootstrap.cc:654] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:21.789456 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 173150us
I20260317 16:18:21.900599 20967 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:21.909418 20827 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.971148 20840 consensus_queue.cc:1048] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260317 16:18:21.961913 20968 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.985607 20978 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:21.986208 20925 consensus_queue.cc:237] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:21.988181 20960 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.937950 20938 raft_consensus.cc:2804] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:21.953233 20869 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:21.935195 20050 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
W20260317 16:18:21.989797 20269 consensus_meta.cc:287] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.611s user 0.013s sys 0.002s
W20260317 16:18:21.951810 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 286696us
I20260317 16:18:21.990060 20967 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:21.997393 20978 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:22.027186 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:22.037207 20822 consensus_meta.cc:287] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.967s user 0.006s sys 0.015s
I20260317 16:18:22.022866 20938 raft_consensus.cc:697] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:22.051962 20236 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.023325 20269 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:21.998102 20827 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.055547 20898 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:22.067346 20898 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.067692 20693 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.067847 20898 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.017305 20869 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.072781 20869 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.073467 20771 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.075230 20938 consensus_queue.cc:237] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.047469 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.018193 20960 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.056799 20822 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.008253 20968 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.077626 20891 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.104202 20922 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.100278 20840 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:22.108336 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 307634us
I20260317 16:18:22.084658 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.037508 20978 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.110354 20199 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:22.062074 20318 tablet_bootstrap.cc:654] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:22.083079 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.075241 20827 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.087625 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.120935 20091 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:22.115475 20826 consensus_meta.cc:287] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.593s user 0.013s sys 0.000s
W20260317 16:18:22.177583 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 63517.4us
W20260317 16:18:22.206218 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 85568.5us
W20260317 16:18:22.169152 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:22.207331 20238 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:22.116195 20836 consensus_meta.cc:287] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.755s user 0.006s sys 0.006s
I20260317 16:18:22.204941 20264 raft_consensus.cc:2393] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:22.246834 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:22.108850 20943 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260317 16:18:22.250988 20536 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:22.236605 20269 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:22.253237 20960 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.099563 20162 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:22.273574 20016 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:22.112051 20840 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.264997 20700 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.239472 20047 raft_consensus.cc:2368] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:22.195951 20811 consensus_queue.cc:1048] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.080s
I20260317 16:18:22.218183 20273 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:22.283631 20536 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.275395 20869 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.283946 20771 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.237599 20826 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.288715 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.287111 20116 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:22.300343 20822 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.294723 20968 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.295797 20840 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:22.298658 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.295315 20892 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:22.357424 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 138307us
W20260317 16:18:22.291536 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 351565us
I20260317 16:18:22.339499 20827 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:22.297066 20967 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.301265 20700 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:22.364585 20123 consensus_meta.cc:287] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.898s user 0.010s sys 0.007s
I20260317 16:18:22.298343 20980 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260317 16:18:22.368134 20199 raft_consensus.cc:1240] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260317 16:18:22.336385 20536 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.366307 20751 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.357885 20771 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election won for term 2
W20260317 16:18:22.356263 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 256819us
I20260317 16:18:22.340377 20869 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.368638 20272 raft_consensus.cc:2468] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:22.378815 20195 raft_consensus.cc:2468] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:22.408103 20968 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:22.411090 20968 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:22.411600 20267 consensus_meta.cc:287] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.646s user 0.012s sys 0.000s
I20260317 16:18:22.412030 20267 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:22.413110 20968 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.413196 20993 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:22.413858 20993 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.416072 20993 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.417304 20123 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 107) took 1136 ms. Trace:
I20260317 16:18:22.419248 20869 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.402657 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.420161 20266 raft_consensus.cc:1275] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:22.422755 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:22.438588 20050 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:22.439078 20198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 152) took 1491 ms. Trace:
W20260317 16:18:22.438709 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:22.391809 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 344676us
I20260317 16:18:22.439916 20198 rpcz_store.cc:276] 0317 16:18:20.947573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.948851 (+ 1278us) service_pool.cc:225] Handling call
0317 16:18:20.949242 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:21.601419 (+652177us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:22.438799 (+837380us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b540005b4d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:22.439016 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1277747584}
I20260317 16:18:22.377113 20892 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:22.406301 20916 consensus_meta.cc:287] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.593s user 0.000s sys 0.011s
I20260317 16:18:22.442613 20916 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.427026 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:22.444669 20269 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.445609 20916 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:22.430004 20016 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:22.426224 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 127259us
I20260317 16:18:22.435321 20827 consensus_queue.cc:1048] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:22.436424 20967 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:22.424700 20988 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.437211 20273 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:22.449275 20236 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.451076 20320 tablet_bootstrap.cc:492] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:22.455888 20320 ts_tablet_manager.cc:1403] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.849s user 0.007s sys 0.020s
I20260317 16:18:22.424459 20321 tablet_bootstrap.cc:492] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:22.457027 20321 ts_tablet_manager.cc:1403] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.068s user 0.012s sys 0.029s
I20260317 16:18:22.437845 20700 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:22.395308 20015 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:22.459314 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:22.460511 20892 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.461961 20016 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.396858 20751 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:22.449011 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 81231.5us
I20260317 16:18:22.479205 20322 tablet_bootstrap.cc:492] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:22.500499 20165 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.500602 20322 ts_tablet_manager.cc:1403] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.919s user 0.023s sys 0.006s
I20260317 16:18:22.497911 20845 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.506726 21002 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:22.463016 20088 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:22.507324 21002 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.428220 20123 rpcz_store.cc:276] 0317 16:18:21.280429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.282013 (+ 1584us) service_pool.cc:225] Handling call
0317 16:18:21.303197 (+ 21184us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:22.408627 (+1105430us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:22.417241 (+ 8614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:22.507727 21002 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.508358 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:22.509092 20268 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.503316 20967 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:22.461433 20162 leader_election.cc:343] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:22.470285 20811 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:22.454591 20916 raft_consensus.cc:2804] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:22.467339 20891 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.529551 20811 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.472963 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:22.475512 20273 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.503526 20822 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:22.529735 20916 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.472939 20986 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.534363 20811 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.499109 20869 consensus_queue.cc:1059] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260317 16:18:22.537191 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.537286 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:22.537299 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:22.537338 20751 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:22.530498 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 76116.9us
I20260317 16:18:22.467427 20988 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.540207 20322 raft_consensus.cc:359] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.569504 20322 raft_consensus.cc:385] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:22.557155 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:22.549419 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:22.498519 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:22.570330 20322 raft_consensus.cc:740] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:22.534512 20973 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:22.527294 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 145711us
I20260317 16:18:22.529569 20771 raft_consensus.cc:697] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:22.542171 20822 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.543438 20916 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:22.545796 20826 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.527830 20898 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.546221 20050 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:22.577021 20117 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.549101 20986 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:22.548635 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 107339us
I20260317 16:18:22.636740 20320 raft_consensus.cc:359] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.612854 20238 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.598624 20322 consensus_queue.cc:260] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.603672 20321 raft_consensus.cc:359] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.669354 20320 raft_consensus.cc:385] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:22.580763 20990 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:22.670025 20321 raft_consensus.cc:385] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:22.691295 20321 raft_consensus.cc:740] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:22.692416 20321 consensus_queue.cc:260] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.612155 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.675657 20871 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.695403 20198 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:22.610394 20967 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.696585 20967 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.699029 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 103) took 1416 ms. Trace:
I20260317 16:18:22.699298 20967 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.598901 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.700160 20195 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:22.603991 20938 consensus_queue.cc:1048] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.078s
I20260317 16:18:22.682318 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.595824 20121 raft_consensus.cc:2468] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:22.653945 20751 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:22.596290 20897 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.645396 20960 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.677047 20269 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:22.683553 20191 raft_consensus.cc:1240] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:22.682674 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.607033 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:22.684340 20165 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:22.671190 20236 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:22.611704 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:22.610656 20983 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:22.690122 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 77354.1us
I20260317 16:18:22.670113 20320 raft_consensus.cc:740] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:22.685182 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.659435 21002 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:22.644001 20978 consensus_meta.cc:287] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.000s
I20260317 16:18:22.595296 20272 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:22.710716 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260317 16:18:22.679909 20948 log.cc:927] Time spent T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 1.390s user 0.025s sys 0.023s
I20260317 16:18:22.602746 20822 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.716704 20321 ts_tablet_manager.cc:1434] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.259s user 0.000s sys 0.011s
I20260317 16:18:22.601871 20988 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.673643 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:22.698411 20273 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:22.612334 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.735009 20318 tablet_bootstrap.cc:492] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:22.795902 20318 ts_tablet_manager.cc:1403] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 1.122s user 0.018s sys 0.013s
I20260317 16:18:22.756049 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:22.735949 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 223271us
I20260317 16:18:22.706458 20994 raft_consensus.cc:2804] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:22.737447 21006 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:22.737761 20267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 72) took 1713 ms (client timeout 2002 ms). Trace:
I20260317 16:18:22.737572 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:22.737946 20892 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.738590 20271 raft_consensus.cc:1275] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260317 16:18:22.739180 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:22.739630 20973 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:22.821857 20318 raft_consensus.cc:359] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.827498 20318 raft_consensus.cc:385] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:22.731024 20887 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:22.752876 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 81770.2us
I20260317 16:18:22.831959 20238 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.702129 20990 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.754021 20322 ts_tablet_manager.cc:1434] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.248s user 0.008s sys 0.003s
I20260317 16:18:22.754518 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.755769 20320 consensus_queue.cc:260] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:22.839004 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 84574us
W20260317 16:18:22.841099 20016 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:22.606422 20771 consensus_queue.cc:237] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.734067 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:22.761334 20998 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:22.699470 20045 rpcz_store.cc:276] 0317 16:18:21.282704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.288295 (+ 5591us) service_pool.cc:225] Handling call
0317 16:18:22.103917 (+815622us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:22.104151 (+ 234us) raft_consensus.cc:1471] Updating replica for 2 ops
0317 16:18:22.104568 (+ 417us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:22.104598 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0317 16:18:22.142237 (+ 37639us) log.cc:844] Serialized 52 byte log entry
0317 16:18:22.390564 (+248327us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:22.395301 (+ 4737us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:22.403019 (+ 7718us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:22.664220 (+261201us) raft_consensus.cc:1685] finished
0317 16:18:22.664241 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:22.698962 (+ 34721us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3252,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":3191,"spinlock_wait_cycles":13162368,"thread_start_us":124925,"threads_started":1,"wal-append.queue_time_us":304272}
I20260317 16:18:22.732275 20960 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.753130 20968 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:22.787142 20239 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:22.812393 20198 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:22.803728 20536 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:22.861696 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 97874.5us
I20260317 16:18:22.746821 20938 raft_consensus.cc:2804] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:22.714454 20751 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.729652 20897 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.735775 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:22.799921 20165 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:22.814605 20892 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.816649 20988 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:22.823542 20692 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.824249 20836 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:22.886294 21002 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.886643 20836 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.742226 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:22.804029 20994 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.797878 20268 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:22.808799 20190 raft_consensus.cc:1275] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:22.787346 20978 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.850899 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:22.819684 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:22.859404 20322 tablet_bootstrap.cc:492] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:22.869422 20898 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:22.870285 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 70498.6us
I20260317 16:18:22.811265 20271 pending_rounds.cc:85] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: Aborting all ops after (but not including) 1
I20260317 16:18:22.870210 20049 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:22.848734 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.874028 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 160) took 1913 ms. Trace:
I20260317 16:18:22.882329 20320 ts_tablet_manager.cc:1434] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.426s user 0.014s sys 0.003s
I20260317 16:18:22.858824 20998 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.825245 20973 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.841215 20887 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.865517 20122 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:22.885594 20988 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.828141 20318 raft_consensus.cc:740] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:22.903180 21002 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:22.891578 20043 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:22.865101 20869 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:22.809090 20267 rpcz_store.cc:269] 0317 16:18:21.023947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.026131 (+ 2184us) service_pool.cc:225] Handling call
0317 16:18:21.256447 (+230316us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:22.737579 (+1481132us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54006842d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:22.737692 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1065136128}
I20260317 16:18:22.867213 20938 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:22.859087 20042 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:22.866355 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:22.853363 20269 raft_consensus.cc:2393] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:22.888125 20041 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.879597 20827 raft_consensus.cc:2804] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:22.896154 20994 raft_consensus.cc:3060] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:22.964610 20751 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:22.896610 21010 log.cc:927] Time spent T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260317 16:18:22.860291 20986 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:22.903841 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 53095.3us
I20260317 16:18:22.917322 20271 pending_rounds.cc:107] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260317 16:18:22.928542 20628 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:22.929039 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.929739 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:22.930558 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:22.891736 20943 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:22.866010 20968 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.877406 21011 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:22.901938 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:22.759387 20983 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:22.945091 20318 consensus_queue.cc:260] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.830525 21006 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:22.892223 20692 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:22.950381 20320 tablet_bootstrap.cc:492] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:22.906718 20264 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:22.929809 20836 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.952884 21005 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.932875 20898 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:22.937062 21013 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:22.937367 20118 raft_consensus.cc:1275] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:22.916723 20893 consensus_queue.cc:1059] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260317 16:18:22.935155 20998 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:22.937870 20826 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.963429 20990 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:22.918833 20853 consensus_queue.cc:1048] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260317 16:18:22.946661 21000 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:22.957943 20938 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:22.981801 20973 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.966028 20827 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:22.991868 21009 log.cc:927] Time spent T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20260317 16:18:22.991752 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:22.970686 20845 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:22.993673 21001 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.996879 20887 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:22.997411 20194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 157) took 1388 ms. Trace:
I20260317 16:18:22.998890 20988 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:22.999398 20322 tablet_bootstrap.cc:654] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:22.965003 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:22.951553 21002 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:22.961685 20192 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:22.976259 20271 raft_consensus.cc:2889] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:22.969192 21016 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.989470 20871 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:22.991773 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:22.985157 20628 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:23.003014 20943 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:22.939915 20015 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:22.988060 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:23.010000 20045 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:22.992961 20088 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:23.013480 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:23.043998 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 53705.3us
I20260317 16:18:23.415292 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 81) took 1359 ms. Trace:
I20260317 16:18:23.049786 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:23.294996 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:23.091310 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:23.053256 20318 ts_tablet_manager.cc:1434] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.257s user 0.007s sys 0.001s
I20260317 16:18:23.522709 20320 tablet_bootstrap.cc:654] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:23.074024 20536 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:23.704748 20746 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.705909 20746 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:23.593706 20042 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.001s sys 0.001s
I20260317 16:18:23.706907 20612 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:23.707623 20042 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.001s sys 0.002s
I20260317 16:18:23.708240 20612 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:23.709684 20746 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:23.711647 20826 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:23.712561 20628 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:23.713583 20628 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.713765 20693 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:23.645951 20165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.123.196:43213 timed out after 2.002s (SENT) [suppressed 16 similar messages]
I20260317 16:18:23.665426 20197 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:23.640686 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:23.677323 20120 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:23.702739 20318 ts_tablet_manager.cc:1467] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Tablet startup took 906ms
W20260317 16:18:23.445935 20822 consensus_meta.cc:287] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.639s user 0.012s sys 0.002s
I20260317 16:18:23.721527 20822 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:23.717355 20771 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:22.919337 20046 rpcz_store.cc:276] 0317 16:18:20.960905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.963395 (+ 2490us) service_pool.cc:225] Handling call
0317 16:18:20.997980 (+ 34585us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:21.031219 (+ 33239us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:21.031260 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:21.031877 (+ 617us) log.cc:844] Serialized 26 byte log entry
0317 16:18:21.186850 (+154973us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:21.186942 (+ 92us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:21.187063 (+ 121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:22.679162 (+1492099us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:22.840244 (+161082us) raft_consensus.cc:1685] finished
0317 16:18:22.840272 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:22.870225 (+ 29953us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43,"raft.queue_time_us":11817,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":21060,"spinlock_wait_cycles":9976192,"thread_start_us":97463,"threads_started":1,"wal-append.queue_time_us":221500}
W20260317 16:18:23.523556 20811 consensus_meta.cc:287] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.974s user 0.007s sys 0.008s
W20260317 16:18:23.600832 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 664111us
W20260317 16:18:23.601944 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 610709us
I20260317 16:18:23.725414 20811 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:23.726433 20015 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:23.639078 20117 consensus_meta.cc:287] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.950s user 0.012s sys 0.000s
I20260317 16:18:23.714624 20693 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:23.727698 20117 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
W20260317 16:18:23.640906 20163 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.005s sys 0.005s
I20260317 16:18:23.728812 20117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 145) took 1204 ms. Trace:
W20260317 16:18:23.729015 20163 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.006s sys 0.006s
I20260317 16:18:23.729269 20117 rpcz_store.cc:276] 0317 16:18:22.525206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.526727 (+ 1521us) service_pool.cc:225] Handling call
0317 16:18:23.728688 (+1201961us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54144}
I20260317 16:18:23.312768 20692 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:23.643357 20272 consensus_meta.cc:287] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.850s user 0.009s sys 0.004s
I20260317 16:18:23.731343 20692 raft_consensus.cc:2749] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:23.731617 20272 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
W20260317 16:18:23.720726 20318 ts_tablet_manager.cc:1469] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Trace:
0317 16:18:04.448548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.448711 (+ 163us) service_pool.cc:225] Handling call
0317 16:18:04.551404 (+102693us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.551496 (+ 92us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.848762 (+297266us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:04.968279 (+119517us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:04.968794 (+ 515us) inbound_call.cc:173] Queueing success response
0317 16:18:21.669637 (+16700843us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:22.796212 (+1126575us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:22.799136 (+ 2924us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:22.799715 (+ 579us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":10016,"raft.queue_time_us":2427,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":7128,"tablet-open.queue_time_us":16700944}
I20260317 16:18:23.732424 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 131) took 1284 ms. Trace:
I20260317 16:18:23.663620 20194 rpcz_store.cc:276] 0317 16:18:21.609323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.627173 (+ 17850us) service_pool.cc:225] Handling call
0317 16:18:21.665700 (+ 38527us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:21.670862 (+ 5162us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:21.672799 (+ 1937us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:21.766963 (+ 94164us) log.cc:844] Serialized 26 byte log entry
0317 16:18:22.053586 (+286623us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:22.129043 (+ 75457us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:22.201673 (+ 72630us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:22.964352 (+762679us) spinlock_profiling.cc:243] Waited 62 ms on lock 0x7b5400537620. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39275fee 00007fea39280b60 00007fea39280b08 00007fea39280a88 00007fea39280a43 00007fea3927fccc 00007fea3f142b26 00007fea3f13e278 00007fea392bc6fb 00007fea392bf1f0
0317 16:18:22.965807 (+ 1455us) raft_consensus.cc:1685] finished
0317 16:18:22.969038 (+ 3231us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:22.997337 (+ 28299us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88,"spinlock_wait_cycles":142524672,"thread_start_us":129822,"threads_started":1,"wal-append.queue_time_us":294667}
I20260317 16:18:23.671015 20196 raft_consensus.cc:2393] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:23.734310 20612 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:23.639592 21034 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:23.735070 20612 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.735559 21034 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:23.735599 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:23.735895 20612 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:23.736325 21034 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:23.714566 20826 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:23.594094 20044 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.000s sys 0.001s
W20260317 16:18:23.738679 20044 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.001s sys 0.001s
I20260317 16:18:23.601774 20043 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:23.739775 20043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 113) took 1005 ms. Trace:
I20260317 16:18:23.740232 20044 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 161) took 2457 ms. Trace:
I20260317 16:18:23.724076 20771 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:23.731108 20088 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:23.740248 20043 rpcz_store.cc:276] 0317 16:18:22.734218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.961227 (+227009us) service_pool.cc:225] Handling call
0317 16:18:23.739675 (+778448us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:23.747303 20811 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:23.732954 20272 rpcz_store.cc:276] 0317 16:18:22.447914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.450095 (+ 2181us) service_pool.cc:225] Handling call
0317 16:18:23.732299 (+1282204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:23.200650 20869 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.767023 20871 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:23.768064 20869 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:23.768631 20853 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.769482 20853 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:23.770828 20871 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:23.771071 20869 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:23.771628 20871 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:23.772009 20853 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:23.740773 20044 rpcz_store.cc:276] 0317 16:18:21.282400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.289812 (+ 7412us) service_pool.cc:225] Handling call
0317 16:18:22.819709 (+1529897us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:22.819967 (+ 258us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:22.917075 (+ 97108us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:22.917133 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:22.938009 (+ 20876us) log.cc:844] Serialized 26 byte log entry
0317 16:18:23.739426 (+801417us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:23.739541 (+ 115us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:23.739647 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:23.739760 (+ 113us) raft_consensus.cc:1685] finished
0317 16:18:23.739784 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:23.740142 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":383,"raft.queue_time_us":35367,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":4538,"spinlock_wait_cycles":13673856,"thread_start_us":846634,"threads_started":2,"wal-append.queue_time_us":557224}
I20260317 16:18:23.777341 20191 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:23.751698 20612 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:23.759562 20827 raft_consensus.cc:3060] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:23.818415 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 172720us
I20260317 16:18:23.714615 20628 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:23.018582 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:23.909144 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:23.911201 20943 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:23.913640 20693 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:23.419548 20916 consensus_meta.cc:287] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.772s user 0.011s sys 0.003s
W20260317 16:18:23.096789 21018 log.cc:927] Time spent T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260317 16:18:23.643819 20270 rpcz_store.cc:276] 0317 16:18:22.056395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.445072 (+388677us) service_pool.cc:225] Handling call
0317 16:18:22.445472 (+ 400us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:22.446085 (+ 613us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:22.446118 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:22.446647 (+ 529us) log.cc:844] Serialized 26 byte log entry
0317 16:18:22.728304 (+281657us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:22.728357 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:22.743446 (+ 15089us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:23.414850 (+671404us) raft_consensus.cc:1685] finished
0317 16:18:23.414910 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:23.415172 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2054,"raft.queue_time_us":1519,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":3246,"thread_start_us":281166,"threads_started":1,"wal-append.queue_time_us":344888}
I20260317 16:18:23.819751 20822 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:23.813019 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 86985.3us
I20260317 16:18:23.340149 21022 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:23.089581 20992 log.cc:927] Time spent T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.662s user 0.004s sys 0.000s
I20260317 16:18:23.926400 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:23.972605 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 114) took 1128 ms. Trace:
I20260317 16:18:23.973145 20116 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 4
I20260317 16:18:23.975013 21027 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.971932 20916 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:23.975800 21027 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:23.415949 20968 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:23.973170 20042 rpcz_store.cc:276] 0317 16:18:22.843995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.980528 (+136533us) service_pool.cc:225] Handling call
0317 16:18:22.994420 (+ 13892us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:22.994863 (+ 443us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:22.994903 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:22.995546 (+ 643us) log.cc:844] Serialized 26 byte log entry
0317 16:18:23.708852 (+713306us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:23.708943 (+ 91us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:23.709118 (+ 175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:23.972086 (+262968us) raft_consensus.cc:1685] finished
0317 16:18:23.972120 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:23.972482 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":300,"raft.queue_time_us":6672,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":5085,"thread_start_us":712437,"threads_started":1,"wal-append.queue_time_us":443693}
I20260317 16:18:23.928274 20893 raft_consensus.cc:3055] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:23.929363 20267 raft_consensus.cc:2393] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:23.929171 21021 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:23.922989 20612 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:23.930150 21000 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:23.930999 20898 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:23.931774 20269 raft_consensus.cc:2368] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4.
I20260317 16:18:23.896406 20871 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:23.932345 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 253423us
I20260317 16:18:24.029400 21016 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.033185 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.034147 20124 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:23.896076 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.034942 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 148) took 1367 ms. Trace:
I20260317 16:18:24.035358 20124 rpcz_store.cc:276] 0317 16:18:22.667094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.761346 (+ 94252us) service_pool.cc:225] Handling call
0317 16:18:24.029876 (+1268530us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.034743 (+ 4867us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68208384}
I20260317 16:18:23.047981 20978 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:23.131893 20892 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:23.955082 20701 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.957384 21037 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:23.958196 20118 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 112) took 1038 ms. Trace:
I20260317 16:18:23.957379 20045 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:24.082257 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 173) took 1240 ms. Trace:
I20260317 16:18:23.031405 20960 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:23.963157 21001 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:23.339803 21006 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:23.050275 21019 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.011291 21011 raft_consensus.cc:697] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:23.311880 21017 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:23.040179 20983 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:23.532419 21031 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:23.926007 20238 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.001s sys 0.018s
I20260317 16:18:24.025064 21013 raft_consensus.cc:697] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:24.102118 20238 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.001s sys 0.019s
I20260317 16:18:24.020823 20917 consensus_queue.cc:1048] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260317 16:18:24.011020 20612 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:24.022210 20771 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.022802 20267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 138) took 1055 ms. Trace:
I20260317 16:18:23.050953 20897 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.024566 21022 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.029260 20269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 139) took 1042 ms. Trace:
I20260317 16:18:23.966536 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:23.048349 21005 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.035416 20194 raft_consensus.cc:2393] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:24.041288 20898 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.042542 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 128) took 1326 ms. Trace:
I20260317 16:18:24.043344 20628 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.055272 20827 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:23.995936 20893 raft_consensus.cc:740] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:24.015127 20120 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:24.035235 21016 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.160792 20845 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:24.161412 20845 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:24.162343 20192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34608 (request call id 111) took 1480 ms. Trace:
I20260317 16:18:24.082715 20045 rpcz_store.cc:276] 0317 16:18:22.841977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.871263 (+ 29286us) service_pool.cc:225] Handling call
0317 16:18:24.082150 (+1210887us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:24.052031 20122 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46060 (request call id 111) took 1516 ms. Trace:
I20260317 16:18:24.098110 20700 raft_consensus.cc:3055] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:24.099365 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.026096 21027 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:23.972204 20994 consensus_meta.cc:287] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 1.007s user 0.015s sys 0.006s
I20260317 16:18:24.100353 20871 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.084235 20693 consensus_queue.cc:1048] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20260317 16:18:24.120385 20612 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.124120 20267 rpcz_store.cc:276] 0317 16:18:22.967108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.991457 (+ 24349us) service_pool.cc:225] Handling call
0317 16:18:24.022650 (+1031193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:24.080009 20118 rpcz_store.cc:276] 0317 16:18:22.919826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.728367 (+808541us) service_pool.cc:225] Handling call
0317 16:18:23.729098 (+ 731us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:23.729558 (+ 460us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:23.756541 (+ 26983us) log.cc:844] Serialized 14 byte log entry
0317 16:18:23.920798 (+164257us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:23.935914 (+ 15116us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:23.957759 (+ 21845us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:23.957889 (+ 130us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:23.958105 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":26831,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":97,"raft.queue_time_us":22396,"raft.run_cpu_time_us":487,"raft.run_wall_time_us":485,"thread_start_us":170407,"threads_started":2,"wal-append.queue_time_us":325569}
W20260317 16:18:24.141305 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 843449us
I20260317 16:18:24.085837 21034 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.107365 21019 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.155704 20701 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.167485 20700 raft_consensus.cc:740] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:24.162886 20125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 110) took 1572 ms. Trace:
I20260317 16:18:24.168292 20120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 149) took 1266 ms. Trace:
I20260317 16:18:24.169016 20199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 148) took 1156 ms. Trace:
I20260317 16:18:24.139048 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:24.153650 21000 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.167393 20122 rpcz_store.cc:276] 0317 16:18:22.535168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.537011 (+ 1843us) service_pool.cc:225] Handling call
0317 16:18:24.051239 (+1514228us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54002e1b54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.051796 (+ 557us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":721930496}
I20260317 16:18:24.202407 20191 raft_consensus.cc:2410] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:24.202708 21039 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:24.203385 20983 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:24.122982 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 345179us
I20260317 16:18:24.152897 21031 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.137552 21037 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.135643 20269 rpcz_store.cc:276] 0317 16:18:22.987130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.999975 (+ 12845us) service_pool.cc:225] Handling call
0317 16:18:24.029158 (+1029183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:24.127811 20853 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:24.166926 20194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 152) took 1039 ms. Trace:
I20260317 16:18:24.199618 20994 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.163082 20192 rpcz_store.cc:276] 0317 16:18:22.681853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.682015 (+ 162us) service_pool.cc:225] Handling call
0317 16:18:24.161795 (+1479780us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54001c7ad4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.162082 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":589309824}
I20260317 16:18:24.215536 20199 rpcz_store.cc:276] 0317 16:18:23.012736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.012865 (+ 129us) service_pool.cc:225] Handling call
0317 16:18:24.098980 (+1086115us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.168956 (+ 69976us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6183296}
I20260317 16:18:24.228858 21017 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.229035 20853 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.229535 21022 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.100864 21021 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.216821 20916 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.203387 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.218779 21002 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.217428 21001 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.227810 20771 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:24.147634 20048 rpcz_store.cc:276] 0317 16:18:22.715781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.717668 (+ 1887us) service_pool.cc:225] Handling call
0317 16:18:22.737630 (+ 19962us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:22.866683 (+129053us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:22.869251 (+ 2568us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:22.885870 (+ 16619us) log.cc:844] Serialized 26 byte log entry
0317 16:18:22.936882 (+ 51012us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:22.936942 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:22.937055 (+ 113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:24.042029 (+1104974us) raft_consensus.cc:1685] finished
0317 16:18:24.042067 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.042327 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5980,"raft.queue_time_us":86086,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":151495,"threads_started":2,"wal-append.queue_time_us":83753}
I20260317 16:18:24.232273 20917 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:24.215191 20120 rpcz_store.cc:276] 0317 16:18:22.901719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.964720 (+ 63001us) service_pool.cc:225] Handling call
0317 16:18:24.168209 (+1203489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:24.247514 20322 tablet_bootstrap.cc:492] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:24.162526 21013 consensus_queue.cc:237] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.213351 20125 rpcz_store.cc:276] 0317 16:18:22.534019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.534167 (+ 148us) service_pool.cc:225] Handling call
0317 16:18:22.539273 (+ 5106us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:22.581478 (+ 42205us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:22.581519 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:22.582017 (+ 498us) log.cc:844] Serialized 26 byte log entry
0317 16:18:22.685649 (+103632us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:22.685773 (+ 124us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.004166 (+1318393us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b540004f1d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.015462 (+ 11296us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:24.015950 (+ 488us) raft_consensus.cc:1685] finished
0317 16:18:24.015979 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.106602 (+ 90623us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18994,"raft.queue_time_us":21403,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2139,"spinlock_wait_cycles":52345728,"thread_start_us":103293,"threads_started":1,"wal-append.queue_time_us":216473}
I20260317 16:18:24.256880 21005 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.245654 20196 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.244119 20273 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.229758 20194 rpcz_store.cc:276] 0317 16:18:23.124702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.734585 (+609883us) service_pool.cc:225] Handling call
0317 16:18:24.156912 (+422327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:24.262013 20893 consensus_queue.cc:260] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.245831 20983 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.263612 20771 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.263854 20917 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.265645 20983 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.228928 21039 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.263684 20265 raft_consensus.cc:2393] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:24.267457 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.268704 21019 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.270359 20195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 149) took 1243 ms. Trace:
I20260317 16:18:24.270843 20917 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.266798 20771 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.248589 21037 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.298918 21011 consensus_queue.cc:237] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.299556 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.299886 20090 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:24.300551 20121 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 147) took 1719 ms (client timeout 2074 ms). Trace:
W20260317 16:18:24.301004 20121 rpcz_store.cc:269] 0317 16:18:22.581683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.730028 (+148345us) service_pool.cc:225] Handling call
0317 16:18:24.299256 (+1569228us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.300486 (+ 1230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3151232}
I20260317 16:18:24.303098 20190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 146) took 1301 ms. Trace:
W20260317 16:18:24.304080 20265 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 79) took 2628 ms (client timeout 1963 ms). Trace:
W20260317 16:18:24.304735 20265 rpcz_store.cc:269] 0317 16:18:21.675227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.675985 (+ 758us) service_pool.cc:225] Handling call
0317 16:18:22.928800 (+1252815us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.303784 (+1374984us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400775e54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.303969 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2962936448}
I20260317 16:18:24.305840 20318 tablet_bootstrap.cc:492] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:24.307313 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.308357 20116 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
W20260317 16:18:24.287389 21015 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.166s user 0.031s sys 0.063s
I20260317 16:18:24.309324 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 144) took 1785 ms. Trace:
I20260317 16:18:24.310012 20116 rpcz_store.cc:276] 0317 16:18:22.524344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.526614 (+ 2270us) service_pool.cc:225] Handling call
0317 16:18:23.914484 (+1387870us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.309230 (+394746us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11710720}
I20260317 16:18:24.303566 20190 rpcz_store.cc:276] 0317 16:18:23.001909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.007070 (+ 5161us) service_pool.cc:225] Handling call
0317 16:18:23.047159 (+ 40089us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:24.096011 (+1048852us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:24.096063 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:24.096139 (+ 76us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:24.096186 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.302611 (+206425us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54002116d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.302671 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.302887 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2411197824}
W20260317 16:18:24.311278 21015 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.264s user 0.031s sys 0.064s
I20260317 16:18:24.317916 20322 ts_tablet_manager.cc:1403] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 1.464s user 0.077s sys 0.103s
I20260317 16:18:24.321228 20091 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:24.327656 21031 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.327870 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:24.329221 20050 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:24.316242 20827 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.322232 20771 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 16:18:24.347925 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 63338.6us
I20260317 16:18:24.322553 21005 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:24.342010 21021 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.344135 20701 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.345384 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 130) took 2232 ms. Trace:
I20260317 16:18:24.350051 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 115) took 1497 ms. Trace:
I20260317 16:18:24.350647 21001 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:24.351894 20268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34240 (request call id 129) took 1497 ms. Trace:
I20260317 16:18:24.352526 20321 tablet_bootstrap.cc:492] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:24.269946 20700 consensus_queue.cc:260] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.330089 20612 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:24.337960 20041 consensus_meta.cc:287] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 1.338s user 0.015s sys 0.003s
I20260317 16:18:24.359774 20853 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.295377 20195 rpcz_store.cc:276] 0317 16:18:23.026756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.031635 (+ 4879us) service_pool.cc:225] Handling call
0317 16:18:24.104941 (+1073306us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.270289 (+165348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3591040}
I20260317 16:18:24.368252 20090 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:24.368613 21037 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:24.369415 20041 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
W20260317 16:18:24.378108 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 43556.7us
I20260317 16:18:24.369889 20612 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.370412 21001 raft_consensus.cc:697] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:24.333868 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.381352 21006 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:24.382294 21001 consensus_queue.cc:237] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.382887 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.383502 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.384176 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.384411 20193 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:24.384824 20194 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:24.385020 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55456 (request call id 168) took 1376 ms. Trace:
I20260317 16:18:24.362049 20268 rpcz_store.cc:276] 0317 16:18:22.868004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.934648 (+ 66644us) service_pool.cc:225] Handling call
0317 16:18:22.959414 (+ 24766us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:24.042620 (+1083206us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:24.042672 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:24.042800 (+ 128us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:24.042872 (+ 72us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.351170 (+308298us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400394254. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.351222 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.351586 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2486215040}
W20260317 16:18:24.335314 20050 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 125) took 1629 ms (client timeout 2039 ms). Trace:
W20260317 16:18:24.387063 20050 rpcz_store.cc:269] 0317 16:18:22.705679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.709896 (+ 4217us) service_pool.cc:225] Handling call
0317 16:18:24.327551 (+1617655us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.334193 (+ 6642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6106880}
I20260317 16:18:24.373103 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.376695 20917 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:24.388211 21037 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.388980 20701 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:24.388753 20917 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.389884 20701 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.374578 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 74) took 3171 ms. Trace:
I20260317 16:18:24.374840 20119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 146) took 1849 ms. Trace:
I20260317 16:18:24.381331 20827 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:24.387579 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.373294 21017 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.357646 20049 rpcz_store.cc:276] 0317 16:18:22.850104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.605694 (+755590us) service_pool.cc:225] Handling call
0317 16:18:23.606603 (+ 909us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:23.804302 (+197699us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:23.804350 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:23.935271 (+130921us) log.cc:844] Serialized 26 byte log entry
0317 16:18:24.226710 (+291439us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:24.226746 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.226838 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:24.346830 (+119992us) raft_consensus.cc:1685] finished
0317 16:18:24.346860 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.347050 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":70447,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":255,"thread_start_us":452124,"threads_started":2,"wal-append.queue_time_us":344713}
I20260317 16:18:24.376544 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:24.359220 20266 rpcz_store.cc:276] 0317 16:18:22.112413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.113183 (+ 770us) service_pool.cc:225] Handling call
0317 16:18:22.274806 (+161623us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:22.437865 (+163059us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.344887 (+1907022us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54001cc5d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.345198 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276601216}
I20260317 16:18:24.359774 21039 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.368348 20700 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:24.354843 20047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 121) took 2745 ms (client timeout 1693 ms). Trace:
I20260317 16:18:24.380219 21034 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.421161 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:24.391268 20041 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 126) took 1679 ms (client timeout 2060 ms). Trace:
W20260317 16:18:24.428494 20041 rpcz_store.cc:269] 0317 16:18:22.711949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.714007 (+ 2058us) service_pool.cc:225] Handling call
0317 16:18:24.391077 (+1677070us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b540003b1d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.391208 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111531264}
I20260317 16:18:24.428974 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.429760 20121 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:24.430011 20701 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.431059 21037 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.432047 20272 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:24.430943 20265 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:24.432760 20117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 151) took 1459 ms. Trace:
I20260317 16:18:24.395010 20271 rpcz_store.cc:276] 0317 16:18:21.203465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.203644 (+ 179us) service_pool.cc:225] Handling call
0317 16:18:21.203993 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:23.699924 (+2495931us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.374089 (+674165us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540028ebd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.374386 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3524356352}
I20260317 16:18:24.433724 20199 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:24.426576 20827 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.435966 20117 rpcz_store.cc:276] 0317 16:18:22.973728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.934207 (+960479us) service_pool.cc:225] Handling call
0317 16:18:23.934719 (+ 512us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:24.159209 (+224490us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.432485 (+273276us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000ada54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.432667 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384100608}
W20260317 16:18:24.426921 20047 rpcz_store.cc:269] 0317 16:18:21.497969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:21.498169 (+ 200us) service_pool.cc:225] Handling call
0317 16:18:24.219060 (+2720891us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b54001b44d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.243044 (+ 23984us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1417801472}
I20260317 16:18:24.444185 21002 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
W20260317 16:18:24.445247 20264 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 136) took 1849 ms (client timeout 1981 ms). Trace:
I20260317 16:18:24.417907 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:24.445732 20264 rpcz_store.cc:269] 0317 16:18:22.595570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.611958 (+ 16388us) service_pool.cc:225] Handling call
0317 16:18:24.444853 (+1832895us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400418554. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.445046 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235961600}
I20260317 16:18:24.446192 20269 raft_consensus.cc:2468] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:24.447738 20838 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:24.448300 20838 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.449671 20015 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:24.451242 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.451813 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:24.453158 20091 leader_election.cc:400] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4.
I20260317 16:18:24.453307 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.453425 20266 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.456094 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.444936 21002 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.457285 20938 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.457583 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:24.459044 20887 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:24.427110 21005 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.459491 21002 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.459578 20887 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.461990 21005 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.463949 20891 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:24.458437 20268 raft_consensus.cc:2376] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4. Re-sending same reply.
I20260317 16:18:24.411664 20994 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.395253 21006 raft_consensus.cc:697] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:24.457943 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:24.427940 20320 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.026s sys 0.068s
I20260317 16:18:24.470877 21042 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:24.471086 20917 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:24.471487 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:24.438892 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.468828 20197 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:24.472229 21002 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:24.453506 20091 leader_election.cc:403] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:24.442365 20273 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:24.464015 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.441277 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.437525 20827 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.412003 20119 rpcz_store.cc:276] 0317 16:18:22.525742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.526668 (+ 926us) service_pool.cc:225] Handling call
0317 16:18:24.373641 (+1846973us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:24.373967 (+ 326us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:24.374225 (+ 258us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:24.374259 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:24.374346 (+ 87us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:24.374419 (+ 73us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.374503 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.374648 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14733056}
I20260317 16:18:24.385525 20193 rpcz_store.cc:276] 0317 16:18:23.021825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.051277 (+ 29452us) service_pool.cc:225] Handling call
0317 16:18:24.383130 (+1331853us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:24.384947 (+ 1817us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3129728}
I20260317 16:18:24.517844 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.465329 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.492336 21037 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:24.488740 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.491595 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.481067 20838 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:24.529706 20998 consensus_meta.cc:287] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.522s user 0.011s sys 0.003s
I20260317 16:18:24.530961 20973 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4.
I20260317 16:18:24.531129 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.532158 21002 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:24.483575 20320 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.683s user 0.027s sys 0.068s
W20260317 16:18:24.523155 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 70392us
I20260317 16:18:24.537228 20271 raft_consensus.cc:2393] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:24.500324 20322 raft_consensus.cc:359] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.578619 20322 raft_consensus.cc:385] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:24.579048 20322 raft_consensus.cc:740] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:24.536253 21037 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:24.587786 21037 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:24.588665 20322 consensus_queue.cc:260] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:24.568387 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 118864us
I20260317 16:18:24.525206 20891 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.549640 20887 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.569048 20827 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:24.567852 20270 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:24.576547 20264 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:24.606220 20320 tablet_bootstrap.cc:492] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
W20260317 16:18:24.603103 21049 log.cc:927] Time spent T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.070s user 0.000s sys 0.003s
I20260317 16:18:24.527376 21042 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.605144 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:24.587395 20191 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:24.570264 20998 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.594705 20196 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:24.637137 20318 tablet_bootstrap.cc:654] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:24.600332 20267 raft_consensus.cc:2468] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:24.621973 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:24.639840 20320 ts_tablet_manager.cc:1403] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 1.697s user 0.041s sys 0.077s
I20260317 16:18:24.618734 20893 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.645277 20975 consensus_queue.cc:1048] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:24.640587 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.619644 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.626067 20938 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.627701 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:24.676558 20123 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 143) took 2156 ms (client timeout 2584 ms). Trace:
I20260317 16:18:24.683113 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.683732 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.684152 20194 raft_consensus.cc:2393] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:24.685475 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.647918 21002 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.627915 20827 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.699357 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:24.634423 20845 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.692836 20272 raft_consensus.cc:2393] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
W20260317 16:18:24.677088 20123 rpcz_store.cc:269] 0317 16:18:22.520615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.533555 (+ 12940us) service_pool.cc:225] Handling call
0317 16:18:24.631241 (+2097686us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54001e9e54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.676354 (+ 45113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1666342144}
W20260317 16:18:24.716352 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 43956.9us
I20260317 16:18:24.602465 21006 consensus_queue.cc:237] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.719966 20198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 145) took 1713 ms. Trace:
I20260317 16:18:24.720749 20198 rpcz_store.cc:276] 0317 16:18:22.988563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:22.999587 (+ 11024us) service_pool.cc:225] Handling call
0317 16:18:24.694441 (+1694854us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5400662ca8. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0b5b44 00007fea3e2bea67 00007fea3e2be4b7 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:24.694852 (+ 411us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:24.699395 (+ 4543us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:24.699431 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:24.699482 (+ 51us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:24.699544 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:24.699614 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:24.699756 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5582080}
I20260317 16:18:24.724464 20979 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:24.725153 20979 raft_consensus.cc:697] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:24.725544 20197 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
W20260317 16:18:24.720391 21051 log.cc:927] Time spent T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260317 16:18:24.726950 20891 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:24.727475 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 145) took 1618 ms. Trace:
I20260317 16:18:24.720023 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.728060 20273 rpcz_store.cc:276] 0317 16:18:23.108841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:23.778655 (+669814us) service_pool.cc:225] Handling call
0317 16:18:24.727223 (+948568us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54006847d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:24.727393 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":763151104}
I20260317 16:18:24.699539 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.729538 20121 raft_consensus.cc:2410] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:24.730562 20015 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:24.732457 20321 tablet_bootstrap.cc:654] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:24.732249 20015 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:24.722613 20322 ts_tablet_manager.cc:1434] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.397s user 0.016s sys 0.002s
I20260317 16:18:24.726696 20979 consensus_queue.cc:237] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.733655 19948 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46702 (request call id 35) took 6199 ms. Trace:
I20260317 16:18:24.734261 21013 raft_consensus.cc:2749] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:24.734319 20917 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:24.741084 20827 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:24.742568 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.743631 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.744218 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.744374 20119 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:24.745060 20998 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:24.745532 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.746214 20118 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:24.694655 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:24.761705 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.762353 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.762948 20322 tablet_bootstrap.cc:492] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:24.763368 20120 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:24.746934 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.760684 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:24.768530 20116 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.819192 20196 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:24.783007 20015 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:24.860626 20123 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:24.860219 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.861366 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.861939 20924 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:24.862663 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.867092 20165 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:24.868140 20320 raft_consensus.cc:359] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.884042 20320 raft_consensus.cc:385] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:24.884457 20320 raft_consensus.cc:740] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:24.836979 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.816334 20975 consensus_queue.cc:1048] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260317 16:18:24.892141 21042 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.891918 20924 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.899735 20090 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:24.903661 20238 leader_election.cc:343] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:24.911566 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 44682.4us
W20260317 16:18:24.912400 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 76423.7us
I20260317 16:18:24.914402 20827 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:24.914948 20827 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:24.915222 20827 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.916157 20822 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:24.918268 20822 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:24.918097 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:24.918980 20122 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:24.928443 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.919106 20121 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:24.902987 20320 consensus_queue.cc:260] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:24.893309 20845 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:24.931365 20238 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:24.894392 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.937804 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.932320 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:24.914505 20165 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:24.734257 19948 rpcz_store.cc:276] 0317 16:18:18.534217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.534392 (+ 175us) service_pool.cc:225] Handling call
0317 16:18:24.733204 (+6198812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:21.378868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:21.386391 (+ 7523us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:21.386428 (+ 37us) write_op.cc:435] Acquired schema lock
0317 16:18:21.386447 (+ 19us) tablet.cc:661] Decoding operations
0317 16:18:21.387257 (+ 810us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:21.387332 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:21.387359 (+ 27us) tablet.cc:684] Acquiring locks for 6 operations
0317 16:18:21.387854 (+ 495us) tablet.cc:700] Row locks acquired
0317 16:18:21.387874 (+ 20us) write_op.cc:260] PREPARE: finished
0317 16:18:21.388006 (+ 132us) write_op.cc:270] Start()
0317 16:18:21.388110 (+ 104us) write_op.cc:276] Timestamp: P: 1773764301387967 usec, L: 0
0317 16:18:21.388129 (+ 19us) op_driver.cc:348] REPLICATION: starting
0317 16:18:21.388745 (+ 616us) log.cc:844] Serialized 2499 byte log entry
0317 16:18:22.312797 (+924052us) op_driver.cc:464] REPLICATION: finished
0317 16:18:22.760640 (+447843us) write_op.cc:301] APPLY: starting
0317 16:18:22.790503 (+ 29863us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:22.809204 (+ 18701us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:22.842991 (+ 33787us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:22.957441 (+114450us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:22.983384 (+ 25943us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0317 16:18:22.999398 (+ 16014us) write_op.cc:312] APPLY: finished
0317 16:18:23.022161 (+ 22763us) log.cc:844] Serialized 52 byte log entry
0317 16:18:23.466523 (+444362us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:23.502499 (+ 35976us) write_op.cc:454] Released schema lock
0317 16:18:23.527750 (+ 25251us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:23.535624 (+ 7874us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:23.535674 (+ 50us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":405328,"apply.run_cpu_time_us":13826,"apply.run_wall_time_us":784351,"num_ops":6,"prepare.queue_time_us":212293,"prepare.run_cpu_time_us":58278,"prepare.run_wall_time_us":142945,"raft.queue_time_us":549671,"raft.run_cpu_time_us":15664,"raft.run_wall_time_us":326117,"replication_time_us":886378,"thread_start_us":871501,"threads_started":5,"wal-append.queue_time_us":634925,"wal-append.run_cpu_time_us":24359,"wal-append.run_wall_time_us":2171925}]]}
I20260317 16:18:24.954223 20122 raft_consensus.cc:3060] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:24.952008 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:24.898653 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 127943us
I20260317 16:18:24.950471 20891 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:24.960213 20015 leader_election.cc:343] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:24.963377 20015 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:24.965242 20239 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:24.918721 20822 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.966156 21013 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:24.966706 21013 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:24.967190 21013 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:24.968081 20700 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:24.968837 20700 raft_consensus.cc:697] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:24.968650 20827 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:24.970010 20700 consensus_queue.cc:237] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:24.951014 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.939687 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.928465 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:24.951946 20124 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 4: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:24.950906 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:24.965252 20994 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:24.980616 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:24.981294 21013 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:24.982283 20891 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:24.983193 20827 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:24.990350 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 296843us
I20260317 16:18:24.958344 19946 catalog_manager.cc:5671] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 1 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
W20260317 16:18:24.997426 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 66181.7us
W20260317 16:18:25.005249 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 118148us
I20260317 16:18:25.000432 20041 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:24.987756 21052 log.cc:927] Time spent T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.245s user 0.000s sys 0.003s
I20260317 16:18:25.028072 20238 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:25.009521 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 95099us
W20260317 16:18:25.022169 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 55623.8us
I20260317 16:18:25.024801 20922 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:25.031381 20320 ts_tablet_manager.cc:1434] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.359s user 0.010s sys 0.002s
I20260317 16:18:25.032778 20165 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:25.035144 20239 leader_election.cc:343] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:25.039836 20239 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:25.037115 20165 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:25.006980 20845 raft_consensus.cc:2804] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:25.009214 20120 raft_consensus.cc:2468] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:25.065698 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:25.018260 20193 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:25.069703 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:25.084592 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 56612.5us
I20260317 16:18:25.085505 20238 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.086915 20935 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:25.087117 20998 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:25.087661 20998 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:25.088052 20998 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.089794 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.091037 21015 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:25.091248 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.091583 21015 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.091754 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.091914 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.092260 21015 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:25.092471 20042 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:25.092684 20050 raft_consensus.cc:2368] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:25.092896 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.074553 20887 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:25.093338 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.093689 20043 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:25.091279 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.086524 20238 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.094990 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:25.095671 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 55981.6us
W20260317 16:18:25.096643 20239 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:25.097791 20239 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:25.101554 20238 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.042085 20845 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.043138 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.028082 20924 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.070860 20827 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:25.075230 20938 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.109349 20887 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:25.086951 20701 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.042941 20922 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.106477 20845 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:25.134639 20239 leader_election.cc:343] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:25.113511 19946 catalog_manager.cc:5671] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 1 to 2, leader changed from 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195) to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 2 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:25.112890 20827 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.150396 20322 tablet_bootstrap.cc:654] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:25.115594 20122 raft_consensus.cc:2468] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:25.147389 20827 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:25.152416 21036 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:25.152808 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.153208 20015 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:25.154224 20271 raft_consensus.cc:2410] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:25.154403 20320 tablet_bootstrap.cc:492] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:25.154641 20967 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:25.154980 20892 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:25.155236 20967 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.155874 20892 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.156468 20892 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.157392 20967 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.158457 20967 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:25.152439 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.159494 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.160871 20267 raft_consensus.cc:2368] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:25.160887 20967 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.161677 20967 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.168974 20935 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:25.169859 20935 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.176131 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.177845 20042 raft_consensus.cc:2410] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:25.148211 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.193243 20199 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
W20260317 16:18:25.194386 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 46398.9us
W20260317 16:18:25.195406 20090 leader_election.cc:343] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:25.025863 20994 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.136582 20826 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:25.198189 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.211603 20199 raft_consensus.cc:2393] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:25.190215 20822 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.126721 20938 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.191830 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.146065 20701 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.235723 19946 catalog_manager.cc:5671] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b reported cstate change: term changed from 1 to 2, leader changed from 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194) to 9a3afb098d6a45f0b405566771a1157b (127.16.123.193). New cstate: current_term: 2 leader_uuid: "9a3afb098d6a45f0b405566771a1157b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:25.243225 20239 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:25.251425 20239 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.228493 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.248850 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.269917 20122 raft_consensus.cc:3060] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 4
I20260317 16:18:25.270599 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.270634 20701 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:25.277448 21037 consensus_meta.cc:287] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.685s user 0.013s sys 0.000s
I20260317 16:18:25.281777 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.281713 21037 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.237687 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:25.279385 21036 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.280864 20701 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:25.251188 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:25.282701 20320 tablet_bootstrap.cc:654] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:25.295984 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:25.285144 20826 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:25.287494 21036 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.254192 21013 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:25.268764 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:25.261323 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:25.331956 20016 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:25.282716 20193 raft_consensus.cc:2393] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
W20260317 16:18:25.304584 20266 consensus_meta.cc:287] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.848s user 0.006s sys 0.011s
W20260317 16:18:25.337402 20893 consensus_meta.cc:287] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.672s user 0.006s sys 0.010s
I20260317 16:18:25.269953 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.283581 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:25.286639 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.332412 20119 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:25.311445 20267 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:25.364270 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 68354us
I20260317 16:18:25.290126 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.287240 20043 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:25.377529 20840 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:25.381242 20827 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.348425 20826 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.284582 20922 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.306707 20822 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.321569 20938 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:25.290521 20979 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.384821 20318 tablet_bootstrap.cc:492] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:25.384191 20979 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.385390 20318 ts_tablet_manager.cc:1403] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 1.080s user 0.047s sys 0.089s
I20260317 16:18:25.296236 20892 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.335022 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.295714 20936 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260317 16:18:25.386859 20042 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 4: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:25.388516 20046 raft_consensus.cc:2393] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 4: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:25.377599 20191 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:25.386720 20979 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.387125 20892 raft_consensus.cc:2804] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:25.391227 20892 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.391074 20318 raft_consensus.cc:359] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.391654 20892 raft_consensus.cc:3060] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.362138 20917 raft_consensus.cc:2749] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:25.285360 21037 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:25.288116 20701 raft_consensus.cc:697] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:25.301430 20122 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:25.287904 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.364916 20266 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:25.289603 20967 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.292313 20998 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.376561 20165 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:25.391865 20318 raft_consensus.cc:385] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:18:25.413682 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:25.433972 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 183012us
W20260317 16:18:25.435437 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 58968.8us
I20260317 16:18:25.421612 20318 raft_consensus.cc:740] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:25.409579 20893 raft_consensus.cc:2804] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:25.444274 20165 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:25.416069 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 314693us
I20260317 16:18:25.388566 20041 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:25.434592 20994 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.457365 20321 tablet_bootstrap.cc:492] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
W20260317 16:18:25.405864 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 252729us
I20260317 16:18:25.453295 20318 consensus_queue.cc:260] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.392326 20197 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:25.438350 21002 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:25.469961 20321 ts_tablet_manager.cc:1403] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.118s user 0.023s sys 0.043s
I20260317 16:18:25.448772 20887 raft_consensus.cc:697] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:25.495560 20091 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:25.476486 20938 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:25.474009 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.510995 21005 consensus_queue.cc:1059] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260317 16:18:25.475661 20700 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.514294 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.493203 20826 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.490888 20893 raft_consensus.cc:697] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:25.556092 20893 consensus_queue.cc:237] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.557271 20892 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.558139 20191 raft_consensus.cc:1240] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260317 16:18:25.541340 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:25.493944 20994 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:25.602597 20322 tablet_bootstrap.cc:492] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:25.603186 20994 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.604621 20322 ts_tablet_manager.cc:1403] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.842s user 0.026s sys 0.000s
I20260317 16:18:25.515643 21006 raft_consensus.cc:2804] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:25.528865 20116 consensus_meta.cc:287] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.681s user 0.017s sys 0.005s
I20260317 16:18:25.496763 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.525161 20845 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.551558 20700 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.557721 20935 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.629117 20935 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.559815 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.630048 21005 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.630918 20197 raft_consensus.cc:2393] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:25.632125 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.631294 21005 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.632460 20268 raft_consensus.cc:1275] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:25.561839 20924 raft_consensus.cc:3055] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:25.633117 20924 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:25.562223 20826 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.608014 21002 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.562572 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.477305 20973 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260317 16:18:25.562919 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.613173 20116 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:25.622182 20700 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.634104 20924 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.563442 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.562575 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:25.529011 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 84808.6us
I20260317 16:18:25.545394 20122 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:25.557294 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.571309 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.533797 21015 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.479719 20701 consensus_queue.cc:237] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.574378 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.575361 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.576078 20043 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:25.577342 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.536350 20827 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:25.540861 20938 raft_consensus.cc:697] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:25.580312 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.580672 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.581200 20998 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:25.587940 20967 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.606231 20994 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:25.467990 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 373023us
I20260317 16:18:25.610184 20318 ts_tablet_manager.cc:1434] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.224s user 0.013s sys 0.001s
I20260317 16:18:25.544194 20887 consensus_queue.cc:237] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.581213 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.636085 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.655968 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:25.637707 20199 raft_consensus.cc:2393] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:25.641139 20165 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.612494 21006 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.695248 21005 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:25.755080 21073 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.635911 21068 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.715996 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:25.697144 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 201692us
I20260317 16:18:25.637427 20924 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.634104 20892 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:25.631541 20845 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:25.781476 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 64740.5us
I20260317 16:18:25.782224 20318 tablet_bootstrap.cc:492] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
W20260317 16:18:25.783149 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 127304us
I20260317 16:18:25.635710 20119 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
W20260317 16:18:25.785194 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 144146us
I20260317 16:18:25.639204 20272 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:25.768939 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:25.789119 20264 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:25.762488 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 45130.9us
I20260317 16:18:25.790135 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:25.741997 21036 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.635401 21002 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.794999 20199 raft_consensus.cc:1240] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:25.782141 20265 raft_consensus.cc:2393] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:25.792870 20194 raft_consensus.cc:2393] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:25.791121 20827 raft_consensus.cc:2804] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:25.791002 20893 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:25.780758 21006 raft_consensus.cc:3060] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.813041 20321 raft_consensus.cc:359] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.832021 20321 raft_consensus.cc:385] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:25.827806 20165 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:25.832337 20321 raft_consensus.cc:740] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:25.836344 20239 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.820096 20998 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:25.839517 21079 log.cc:927] Time spent T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260317 16:18:25.838542 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.841703 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.842387 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.841730 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.845584 20320 tablet_bootstrap.cc:492] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:25.842283 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.846233 20320 ts_tablet_manager.cc:1403] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.693s user 0.007s sys 0.008s
I20260317 16:18:25.847676 20938 consensus_queue.cc:237] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.848680 20271 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260317 16:18:25.823361 21080 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:25.851325 21080 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.852388 20935 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:25.848270 20194 raft_consensus.cc:1240] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:25.794443 21073 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.828874 20827 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:25.859635 20827 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:25.862114 20165 leader_election.cc:400] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:25.862076 21073 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:25.864554 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.839560 20998 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.862820 20165 leader_election.cc:403] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:25.841835 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.836791 20321 consensus_queue.cc:260] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:25.866688 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.876711 20917 consensus_queue.cc:1059] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:25.835013 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.829243 20273 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:25.877463 20975 consensus_queue.cc:1059] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:25.877915 20197 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:25.909845 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:25.878527 20195 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.910979 20917 raft_consensus.cc:3055] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:25.911511 20917 raft_consensus.cc:740] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:25.903483 20196 raft_consensus.cc:3060] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.912667 20917 consensus_queue.cc:260] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.914209 20917 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.903461 20935 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:25.915315 20975 raft_consensus.cc:3055] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:25.878914 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.879248 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.883698 20318 tablet_bootstrap.cc:654] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:25.889531 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:25.901520 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 65288us
I20260317 16:18:25.867916 21015 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:25.921327 20272 raft_consensus.cc:2468] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:25.882490 20267 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:25.904419 20165 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:25.909684 20271 raft_consensus.cc:1275] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260317 16:18:25.923460 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:25.915866 20975 raft_consensus.cc:740] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:25.801419 21068 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:25.942268 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.938089 20935 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:25.943939 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.944622 20124 raft_consensus.cc:3060] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.945817 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.946607 20116 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:25.947243 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:25.947605 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.947888 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.948092 20268 raft_consensus.cc:2368] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4.
I20260317 16:18:25.951454 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:25.952095 20998 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.954439 21080 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:25.943514 20975 consensus_queue.cc:260] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:25.955226 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:25.955627 20975 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.955821 20120 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:25.960284 20321 ts_tablet_manager.cc:1434] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.486s user 0.000s sys 0.010s
I20260317 16:18:25.961473 20321 tablet_bootstrap.cc:492] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:25.961829 20090 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:25.963171 20090 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:25.963285 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.963910 21005 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:25.964135 20264 raft_consensus.cc:2368] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
I20260317 16:18:25.964382 21005 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:25.964769 21005 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.964865 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:25.965785 20265 raft_consensus.cc:2393] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:25.966941 20973 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:25.967105 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.967376 20973 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:25.967797 20973 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:25.968009 20118 raft_consensus.cc:2393] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
W20260317 16:18:25.970749 21077 log.cc:927] Time spent T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.191s user 0.002s sys 0.002s
I20260317 16:18:25.952160 20270 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:25.972864 20269 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260317 16:18:25.927348 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:25.978605 20119 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:25.979477 20936 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:25.916827 21073 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:25.980018 20936 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.980497 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.981935 20935 raft_consensus.cc:697] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:25.982905 20935 consensus_queue.cc:237] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:25.983985 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:25.986999 21036 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:25.936748 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:25.997103 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:25.998088 20826 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:25.998482 20826 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:25.998886 20826 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.001920 21073 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:26.002564 21073 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:26.006676 20165 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:26.016197 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.008404 20322 raft_consensus.cc:359] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.021132 20322 raft_consensus.cc:385] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:26.021404 20322 raft_consensus.cc:740] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:26.023123 20967 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:26.026232 20967 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:26.006008 20936 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:26.030469 21005 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.039908 20120 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:26.041976 21015 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:26.052182 21015 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.053638 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.043996 21002 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:26.022275 20322 consensus_queue.cc:260] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.018565 20199 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
W20260317 16:18:26.056581 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 50568us
I20260317 16:18:26.055692 21015 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:26.062909 20936 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:26.063391 20936 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:26.064079 20936 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:26.065932 20124 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:26.068046 20238 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.068254 21070 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:26.076623 20827 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:26.077239 21070 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:26.088611 20194 raft_consensus.cc:2393] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:26.090380 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.091054 21073 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.091662 21068 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:26.094751 21006 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.095567 20116 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:26.096371 20123 raft_consensus.cc:2393] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 2: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:26.019382 20320 raft_consensus.cc:359] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.097707 20320 raft_consensus.cc:385] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:26.099330 20320 raft_consensus.cc:740] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:26.101404 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.103024 20043 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.105400 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.106246 20041 raft_consensus.cc:2393] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:26.109683 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.109967 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.108670 20165 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:26.112787 20165 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:26.099223 21037 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:26.114400 21037 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.116144 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.115953 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.116991 21037 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:26.115936 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.118172 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.118870 20049 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.108773 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.071450 20193 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:26.120682 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.080853 21070 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:26.123005 21073 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.125228 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.126554 20193 raft_consensus.cc:2393] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:26.126808 20827 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:26.127147 20270 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
W20260317 16:18:26.129237 20088 leader_election.cc:336] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.693s (SENT)
I20260317 16:18:26.130213 20273 raft_consensus.cc:2468] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:26.073204 20195 raft_consensus.cc:2468] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
W20260317 16:18:26.141376 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 73435.9us
I20260317 16:18:26.155673 20043 raft_consensus.cc:1240] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:26.157034 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.141331 20320 consensus_queue.cc:260] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:26.189844 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:26.158883 21005 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:26.216037 21100 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:26.216037 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 103515us
I20260317 16:18:26.128948 20827 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:26.123281 20015 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:26.168773 20088 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:26.219831 20321 tablet_bootstrap.cc:654] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:26.218261 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.158964 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.222939 20322 ts_tablet_manager.cc:1434] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.615s user 0.011s sys 0.000s
W20260317 16:18:26.226112 20322 ts_tablet_manager.cc:1467] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Tablet startup took 617ms
I20260317 16:18:26.165107 21072 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:26.227077 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.227327 21072 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:26.227715 21072 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:26.227974 20047 raft_consensus.cc:2393] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:26.228850 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.229921 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.230508 20045 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.233592 21006 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:26.234043 20936 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.235365 20973 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.166966 20196 raft_consensus.cc:2468] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:26.211344 20967 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:26.237399 20973 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:26.217450 21068 consensus_queue.cc:1048] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260317 16:18:26.165369 21073 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:26.241214 21073 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.243230 21073 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:26.243124 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 121454us
I20260317 16:18:26.243204 20967 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:26.244750 20967 raft_consensus.cc:697] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:26.245100 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.245626 20967 consensus_queue.cc:237] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:26.247133 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.247030 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.247931 20049 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.248191 20042 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.248842 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.249375 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.249476 20044 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.250496 20042 raft_consensus.cc:2393] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:26.251348 20936 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.251533 20048 raft_consensus.cc:1275] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:26.250648 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.252238 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.252368 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.252638 20827 raft_consensus.cc:2749] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:26.255043 20238 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.256313 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.274197 20199 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:26.256596 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:26.258088 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:26.226413 20322 ts_tablet_manager.cc:1469] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Trace:
0317 16:18:04.960603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.961059 (+ 456us) service_pool.cc:225] Handling call
0317 16:18:04.964114 (+ 3055us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.964218 (+ 104us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.276453 (+312235us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.334447 (+ 57994us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.334999 (+ 552us) inbound_call.cc:173] Queueing success response
0317 16:18:24.762149 (+19427150us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:25.608426 (+846277us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:25.974944 (+366518us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:25.975448 (+ 504us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":82746,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":9500,"tablet-open.queue_time_us":19427223}
I20260317 16:18:26.258744 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.275828 20238 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:26.277659 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:26.271198 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 325324us
I20260317 16:18:26.266636 20197 raft_consensus.cc:1240] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:26.266822 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.286900 20049 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.291438 20238 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.226528 21100 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.283834 21073 raft_consensus.cc:2804] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
W20260317 16:18:26.302942 20238 leader_election.cc:343] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:26.307588 20196 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:26.309165 20162 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:26.296175 21068 raft_consensus.cc:2804] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:26.304154 20701 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:26.281641 20044 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:26.312683 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.289685 20124 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:26.302057 20975 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:26.289798 20827 consensus_queue.cc:1048] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260317 16:18:26.305438 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.316490 20120 raft_consensus.cc:1240] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:26.340483 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:26.341765 20938 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:26.342053 20118 raft_consensus.cc:2393] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:26.335927 20123 raft_consensus.cc:1240] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260317 16:18:26.344121 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.331977 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.346869 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.345432 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.347671 20046 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.348315 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.349203 20043 raft_consensus.cc:2410] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260317 16:18:26.321897 20701 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:26.351251 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.351383 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.351850 20041 raft_consensus.cc:2376] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3. Re-sending same reply.
I20260317 16:18:26.352449 20048 raft_consensus.cc:2393] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:26.352890 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.353439 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.354031 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.354979 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.306478 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.355967 20050 raft_consensus.cc:2410] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:26.357018 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.358104 20199 raft_consensus.cc:1275] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:26.358203 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.359243 20119 raft_consensus.cc:1275] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260317 16:18:26.359679 20119 pending_rounds.cc:85] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Aborting all ops after (but not including) 1
I20260317 16:18:26.359906 20119 pending_rounds.cc:107] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260317 16:18:26.360183 20119 raft_consensus.cc:2889] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:26.360713 20045 raft_consensus.cc:2410] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260317 16:18:26.361114 20041 raft_consensus.cc:2368] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:26.361883 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.362367 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.362545 21068 raft_consensus.cc:697] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:26.363200 20701 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:26.363430 20041 raft_consensus.cc:2368] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4.
I20260317 16:18:26.363880 21068 consensus_queue.cc:237] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.366781 20043 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:26.367244 20162 leader_election.cc:400] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:26.369637 20162 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:26.370869 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:26.371552 21100 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.340354 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.301654 20826 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.373373 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.346956 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.383464 20322 tablet_bootstrap.cc:492] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:26.385478 20893 consensus_queue.cc:1048] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:26.383401 20936 consensus_queue.cc:1048] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260317 16:18:26.338652 20975 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.373291 20050 raft_consensus.cc:2376] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3. Re-sending same reply.
I20260317 16:18:26.389397 20162 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:26.362694 20045 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.391850 20199 raft_consensus.cc:1275] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:26.397498 20091 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:26.333854 21073 raft_consensus.cc:697] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:26.376422 20935 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:26.386601 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.422324 20091 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.412705 20973 raft_consensus.cc:2749] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:26.426179 20973 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:26.426698 20973 raft_consensus.cc:697] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:26.426813 20091 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.427129 20998 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:26.427642 20973 consensus_queue.cc:237] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.429553 20826 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.408411 20917 consensus_queue.cc:1059] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260317 16:18:26.427726 20998 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.424623 20320 ts_tablet_manager.cc:1434] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.578s user 0.022s sys 0.004s
I20260317 16:18:26.418155 21002 raft_consensus.cc:2804] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:26.432986 20320 ts_tablet_manager.cc:1467] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: Tablet startup took 586ms
W20260317 16:18:26.432191 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 212826us
I20260317 16:18:26.434720 20318 tablet_bootstrap.cc:492] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:26.435892 20318 ts_tablet_manager.cc:1403] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.660s user 0.019s sys 0.009s
I20260317 16:18:26.420537 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.419991 20042 raft_consensus.cc:2468] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:26.439632 20165 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:26.440797 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.417289 20887 consensus_queue.cc:1048] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260317 16:18:26.450712 20321 tablet_bootstrap.cc:492] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:26.451390 20321 ts_tablet_manager.cc:1403] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.490s user 0.016s sys 0.022s
I20260317 16:18:26.433219 21002 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:26.440869 21006 raft_consensus.cc:2804] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:26.438261 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.454209 20165 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:26.426190 21073 consensus_queue.cc:237] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:26.433334 20320 ts_tablet_manager.cc:1469] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096: Trace:
0317 16:18:05.031569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:05.041910 (+ 10341us) service_pool.cc:225] Handling call
0317 16:18:05.047862 (+ 5952us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:05.047950 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.120410 (+ 72460us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.166527 (+ 46117us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.166980 (+ 453us) inbound_call.cc:173] Queueing success response
0317 16:18:25.147699 (+19980719us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:25.846567 (+698868us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:25.960670 (+114103us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:25.961207 (+ 537us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":46491,"raft.queue_time_us":110847,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":10076,"tablet-open.queue_time_us":19962961,"thread_start_us":250202,"threads_started":1}
I20260317 16:18:26.448825 20044 raft_consensus.cc:1275] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:26.482323 20318 raft_consensus.cc:359] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.491578 20318 raft_consensus.cc:385] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:26.492703 20318 raft_consensus.cc:740] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:26.445026 20118 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:26.493269 20322 tablet_bootstrap.cc:654] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:26.447072 20826 consensus_queue.cc:1059] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
I20260317 16:18:26.491627 20165 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:26.494261 20318 consensus_queue.cc:260] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.476585 21002 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:26.476409 21006 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:26.499694 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.505087 20049 raft_consensus.cc:2393] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:26.502074 20165 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:26.459175 21103 raft_consensus.cc:2804] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election won for term 4
W20260317 16:18:26.503150 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 75033.6us
I20260317 16:18:26.450071 21072 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:26.509137 21094 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:26.493970 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.510797 21072 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:26.511135 20938 raft_consensus.cc:3055] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:26.511600 20938 raft_consensus.cc:740] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:26.506819 20935 raft_consensus.cc:2804] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:26.512465 20935 raft_consensus.cc:697] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:26.512938 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:26.513597 20197 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:26.513501 20935 consensus_queue.cc:237] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.514976 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.516209 20998 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.516245 20321 raft_consensus.cc:359] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.518008 20321 raft_consensus.cc:385] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:26.518289 20321 raft_consensus.cc:740] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:26.498694 21100 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:26.519073 21100 raft_consensus.cc:697] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
W20260317 16:18:26.520164 20162 leader_election.cc:343] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:26.519994 21100 consensus_queue.cc:237] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.521111 20701 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.521881 20162 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:26.522364 20975 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.508440 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:26.506368 21006 raft_consensus.cc:3060] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:26.510825 20318 ts_tablet_manager.cc:1434] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.075s user 0.009s sys 0.000s
I20260317 16:18:26.530949 20318 tablet_bootstrap.cc:492] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:26.531011 20321 consensus_queue.cc:260] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.512529 20938 consensus_queue.cc:260] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.555580 20320 tablet_bootstrap.cc:492] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
W20260317 16:18:26.558614 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 50655.5us
W20260317 16:18:26.572137 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 65394.9us
I20260317 16:18:26.573271 20091 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:26.530849 21037 raft_consensus.cc:3055] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:26.580560 20091 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:26.604195 20701 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:26.556850 21103 raft_consensus.cc:697] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:26.579164 21037 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:26.573989 20046 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:26.608314 20938 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.636042 20321 ts_tablet_manager.cc:1434] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.184s user 0.005s sys 0.004s
I20260317 16:18:26.639880 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.640638 20049 raft_consensus.cc:2410] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260317 16:18:26.635843 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 99196.8us
W20260317 16:18:26.579111 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 57321.1us
I20260317 16:18:26.602140 20924 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:26.645252 20165 leader_election.cc:400] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
I20260317 16:18:26.650357 20165 leader_election.cc:403] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:26.653491 20165 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.600891 20973 consensus_queue.cc:1048] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260317 16:18:26.664623 20321 tablet_bootstrap.cc:492] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:26.622082 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.669495 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.644066 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:26.658486 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:26.677438 21104 log.cc:927] Time spent T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260317 16:18:26.679090 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:26.658398 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:26.652212 20267 raft_consensus.cc:1275] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:26.647121 21094 raft_consensus.cc:697] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:26.656636 21101 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
I20260317 16:18:26.652854 21110 raft_consensus.cc:2764] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260317 16:18:26.702414 21103 consensus_queue.cc:237] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.702986 20980 consensus_queue.cc:1059] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260317 16:18:26.704767 20924 raft_consensus.cc:3055] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:26.705235 20924 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:26.706233 20924 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.667850 21036 raft_consensus.cc:2804] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:26.713053 21036 raft_consensus.cc:697] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:26.713991 21036 consensus_queue.cc:237] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:26.718977 20320 tablet_bootstrap.cc:654] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:26.721084 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 114529us
I20260317 16:18:26.721879 20091 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:26.724252 20091 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:26.726310 20318 tablet_bootstrap.cc:654] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:26.728579 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.734575 20239 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:26.745314 20049 raft_consensus.cc:2368] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
W20260317 16:18:26.747000 20088 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:26.753386 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 142078us
I20260317 16:18:26.759222 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:26.760092 20272 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
W20260317 16:18:26.751873 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:26.689600 20162 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:26.770242 20236 leader_election.cc:343] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:26.753041 21110 raft_consensus.cc:2804] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:26.770735 20236 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:26.772365 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:26.755515 20701 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:26.772074 20088 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:26.757231 20973 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:26.779155 20088 leader_election.cc:400] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4.
I20260317 16:18:26.785696 20088 leader_election.cc:403] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260317 16:18:26.780073 21105 log.cc:927] Time spent T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.176s user 0.003s sys 0.000s
I20260317 16:18:26.782040 20980 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:26.764722 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.808629 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.775705 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:26.776193 20701 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:26.763386 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:26.774158 21110 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:26.811604 21002 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.790671 21119 consensus_queue.cc:1059] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20260317 16:18:26.816825 21112 log.cc:927] Time spent T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
W20260317 16:18:26.817654 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 47451.8us
I20260317 16:18:26.766086 21005 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:26.817981 20267 raft_consensus.cc:1240] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:26.765589 21118 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260317 16:18:26.871023 20236 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:26.818367 20893 raft_consensus.cc:2749] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:26.818933 20322 tablet_bootstrap.cc:492] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:26.818830 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:26.820338 21114 log.cc:927] Time spent T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
W20260317 16:18:26.820245 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 153390us
I20260317 16:18:26.821723 20975 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:26.821679 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:26.809172 21094 consensus_queue.cc:237] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:26.864221 21110 raft_consensus.cc:3060] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:26.820251 20924 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.876703 20124 raft_consensus.cc:1275] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:26.899695 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:26.900743 20267 raft_consensus.cc:2368] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:26.901053 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:26.901675 20118 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:26.901778 20266 raft_consensus.cc:1240] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: []
I20260317 16:18:26.887372 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:26.904335 20199 raft_consensus.cc:1275] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:26.904984 20265 raft_consensus.cc:2368] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:26.844383 20321 tablet_bootstrap.cc:654] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:26.861338 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 82323.1us
I20260317 16:18:26.824575 20935 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:26.842156 20973 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4.
I20260317 16:18:26.851047 21116 raft_consensus.cc:2749] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:26.867292 20272 raft_consensus.cc:1240] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:26.851925 20980 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.875715 20322 ts_tablet_manager.cc:1403] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.538s user 0.025s sys 0.021s
I20260317 16:18:26.843365 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:26.872576 20236 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:26.859864 20701 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:26.861639 20198 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:26.869675 20190 raft_consensus.cc:2393] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:26.726682 21037 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:26.886324 19946 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46676 (request call id 40) took 8645 ms. Trace:
W20260317 16:18:26.887545 20165 leader_election.cc:343] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:26.953208 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 65725.3us
I20260317 16:18:26.889653 20192 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:26.885450 20975 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.891435 20042 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:26.891949 21002 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:26.887499 20197 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:26.895197 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:26.842957 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:26.878747 20887 raft_consensus.cc:2749] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:26.865192 21006 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.877046 20892 consensus_queue.cc:1059] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260317 16:18:26.877425 21005 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:26.947305 21073 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:26.960187 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.979507 20270 raft_consensus.cc:2393] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
W20260317 16:18:26.971822 20045 consensus_meta.cc:287] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.556s user 0.012s sys 0.000s
I20260317 16:18:26.973192 21130 raft_consensus.cc:3055] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:26.967564 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.001894 20272 raft_consensus.cc:2468] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:26.959605 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:26.976722 20826 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:26.953061 19946 rpcz_store.cc:276] 0317 16:18:18.236316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:18.240210 (+ 3894us) service_pool.cc:225] Handling call
0317 16:18:26.881243 (+8641033us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:25.537617 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:25.576491 (+ 38874us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:25.576558 (+ 67us) write_op.cc:435] Acquired schema lock
0317 16:18:25.576581 (+ 23us) tablet.cc:661] Decoding operations
0317 16:18:25.577444 (+ 863us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:25.577868 (+ 424us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:25.577919 (+ 51us) tablet.cc:684] Acquiring locks for 3 operations
0317 16:18:25.587669 (+ 9750us) tablet.cc:700] Row locks acquired
0317 16:18:25.587691 (+ 22us) write_op.cc:260] PREPARE: finished
0317 16:18:25.587892 (+ 201us) write_op.cc:270] Start()
0317 16:18:25.588072 (+ 180us) write_op.cc:276] Timestamp: P: 1773764305587807 usec, L: 0
0317 16:18:25.588108 (+ 36us) op_driver.cc:348] REPLICATION: starting
0317 16:18:25.591544 (+ 3436us) log.cc:844] Serialized 1325 byte log entry
0317 16:18:26.342668 (+751124us) op_driver.cc:464] REPLICATION: finished
0317 16:18:26.526630 (+183962us) write_op.cc:301] APPLY: starting
0317 16:18:26.526721 (+ 91us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:26.527247 (+ 526us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:26.527393 (+ 146us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:26.527934 (+ 541us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:26.528169 (+ 235us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0317 16:18:26.528232 (+ 63us) write_op.cc:312] APPLY: finished
0317 16:18:26.528508 (+ 276us) log.cc:844] Serialized 34 byte log entry
0317 16:18:26.529308 (+ 800us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:26.529397 (+ 89us) write_op.cc:454] Released schema lock
0317 16:18:26.529450 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:26.529841 (+ 391us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:26.529876 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":218,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":160682,"apply.run_cpu_time_us":2267,"apply.run_wall_time_us":3212,"num_ops":3,"prepare.queue_time_us":233394,"prepare.run_cpu_time_us":70106,"prepare.run_wall_time_us":384159,"raft.queue_time_us":228356,"raft.run_cpu_time_us":9176,"raft.run_wall_time_us":95510,"replication_time_us":754312,"thread_start_us":695243,"threads_started":4,"wal-append.queue_time_us":519532}]]}
W20260317 16:18:27.016463 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 143991us
W20260317 16:18:27.017335 20236 leader_election.cc:343] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:27.006920 20088 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.004395 20045 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:27.033316 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.170397 20049 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:27.000057 20917 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:27.027451 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:27.029736 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.011636 20117 raft_consensus.cc:1275] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260317 16:18:27.157800 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:27.156679 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 142948us
I20260317 16:18:27.028523 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:26.968966 21118 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260317 16:18:27.017587 20320 tablet_bootstrap.cc:492] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:27.005138 21130 raft_consensus.cc:740] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:27.164825 20236 leader_election.cc:304] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.179787 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:27.180954 21094 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:27.183728 20091 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.184792 20125 raft_consensus.cc:2393] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:27.185380 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 178584us
I20260317 16:18:27.222384 20318 tablet_bootstrap.cc:492] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
W20260317 16:18:27.164160 20938 consensus_meta.cc:287] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.516s user 0.008s sys 0.015s
I20260317 16:18:27.193655 20049 raft_consensus.cc:2410] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:27.203922 20320 ts_tablet_manager.cc:1403] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.649s user 0.017s sys 0.031s
W20260317 16:18:27.228894 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 45254.9us
W20260317 16:18:27.220453 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 263661us
I20260317 16:18:27.223234 20318 ts_tablet_manager.cc:1403] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.693s user 0.037s sys 0.017s
I20260317 16:18:27.218433 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.186295 20917 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.242192 20049 raft_consensus.cc:1240] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:27.210327 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:27.223280 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:27.236819 20016 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:27.205188 20047 raft_consensus.cc:2376] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4. Re-sending same reply.
I20260317 16:18:27.271617 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.272097 21073 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:27.272698 21073 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.273783 20046 raft_consensus.cc:1275] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:27.231930 20088 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.258674 20322 raft_consensus.cc:359] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:27.239933 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 222628us
I20260317 16:18:27.275281 20236 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:27.274636 20322 raft_consensus.cc:385] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:18:27.262689 20239 leader_election.cc:343] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:27.266355 21131 raft_consensus.cc:2749] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:27.237383 21006 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.276703 20239 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:27.278622 20016 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:27.218747 20975 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:27.270136 20887 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:27.301215 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 69619.3us
I20260317 16:18:27.302533 20088 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:27.304239 20088 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:27.276201 20322 raft_consensus.cc:740] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:27.289075 20016 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:27.297344 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:27.239234 21124 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:27.297204 20090 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:27.240033 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:27.237957 21005 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.229656 21139 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:27.220230 21094 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.295655 21130 consensus_queue.cc:260] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.311677 21141 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:27.312098 20917 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:27.235857 20198 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:27.313535 20318 raft_consensus.cc:359] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.275894 20936 consensus_queue.cc:1048] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20260317 16:18:27.308977 21124 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:27.323809 20048 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:27.307838 20088 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.322300 20016 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:27.322816 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 60322.9us
W20260317 16:18:27.322363 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 47189.5us
I20260317 16:18:27.296943 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:27.335331 20318 raft_consensus.cc:385] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:27.348847 20318 raft_consensus.cc:740] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:27.349566 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:27.350486 20123 raft_consensus.cc:2393] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:27.301805 20967 raft_consensus.cc:2749] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:27.309460 20980 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.317685 20322 consensus_queue.cc:260] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:27.352631 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 44903.5us
I20260317 16:18:27.349723 20318 consensus_queue.cc:260] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.314175 20924 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:27.352874 20239 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:27.353621 20236 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:27.336055 20892 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:27.358122 20195 raft_consensus.cc:1275] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:27.359048 21145 raft_consensus.cc:2749] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:27.342836 20238 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.315196 20973 consensus_queue.cc:1048] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260317 16:18:27.347975 20935 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:27.341430 20118 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:27.362689 21130 raft_consensus.cc:3060] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:27.316098 20887 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.337942 21149 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:27.331398 21131 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:27.362900 20125 raft_consensus.cc:1240] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:27.328861 21139 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.399617 20320 raft_consensus.cc:359] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.400811 20320 raft_consensus.cc:385] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:27.378233 20016 leader_election.cc:400] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4.
I20260317 16:18:27.345430 21124 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:27.383560 20318 ts_tablet_manager.cc:1434] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.138s user 0.005s sys 0.003s
I20260317 16:18:27.360982 20917 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260317 16:18:27.345227 21080 consensus_queue.cc:1175] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d8ed4b12dc654776aa7e64c0fa224321" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260317 16:18:27.411571 20195 raft_consensus.cc:1240] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:27.387452 20322 ts_tablet_manager.cc:1434] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.450s user 0.012s sys 0.007s
W20260317 16:18:27.376381 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:27.366226 20117 raft_consensus.cc:1275] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:27.411728 20198 raft_consensus.cc:1240] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:27.413360 20321 tablet_bootstrap.cc:492] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:27.419571 20321 ts_tablet_manager.cc:1403] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.755s user 0.029s sys 0.037s
I20260317 16:18:27.401206 20320 raft_consensus.cc:740] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:27.420949 20320 consensus_queue.cc:260] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.412283 21133 raft_consensus.cc:2749] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:27.412091 21138 consensus_queue.cc:1048] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260317 16:18:27.424888 21138 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:27.425294 21138 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:27.425693 21138 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:27.412179 21146 raft_consensus.cc:2749] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:27.427341 21146 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:27.427822 21146 raft_consensus.cc:697] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:27.428711 21146 consensus_queue.cc:237] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.402953 20016 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:27.408581 20192 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:27.410651 21143 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260317 16:18:27.370556 21141 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:27.387971 20088 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:27.417572 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 64039.8us
W20260317 16:18:27.439678 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 51739.3us
I20260317 16:18:27.441411 20088 leader_election.cc:400] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:27.441857 20088 leader_election.cc:403] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260317 16:18:27.441658 20236 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:27.433115 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 61177.1us
I20260317 16:18:27.394593 21145 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:27.371492 20892 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.394486 21149 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:27.451995 20236 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:27.447777 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 44876.6us
I20260317 16:18:27.476850 20125 raft_consensus.cc:1240] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:27.411814 21094 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:27.479187 20318 tablet_bootstrap.cc:492] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:27.431654 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.446202 21131 raft_consensus.cc:2804] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:27.397919 20893 consensus_queue.cc:1048] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260317 16:18:27.481914 20322 tablet_bootstrap.cc:492] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:27.478078 20016 leader_election.cc:400] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
W20260317 16:18:27.478478 20236 leader_election.cc:343] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:27.486110 20320 ts_tablet_manager.cc:1434] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.259s user 0.006s sys 0.001s
I20260317 16:18:27.429451 21073 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:27.488217 20016 leader_election.cc:403] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:27.472712 21142 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:27.476105 21149 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:27.484094 20973 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:27.484409 21145 raft_consensus.cc:2764] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260317 16:18:27.451174 21005 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:27.491688 20236 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.450218 20701 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.499480 21134 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 5
I20260317 16:18:27.499002 20016 leader_election.cc:400] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:27.505314 20016 leader_election.cc:403] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260317 16:18:27.495049 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 197924us
I20260317 16:18:27.473647 20917 consensus_queue.cc:1059] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260317 16:18:27.476293 21072 consensus_queue.cc:1048] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.019s
I20260317 16:18:27.483218 20967 consensus_queue.cc:1059] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260317 16:18:27.487401 21141 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:27.509997 20165 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:27.489127 20050 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:27.494603 21119 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:27.516358 20701 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:27.517166 21156 raft_consensus.cc:2749] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:27.550042 21130 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:27.550858 21130 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:27.553262 20318 tablet_bootstrap.cc:654] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:27.552870 21149 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.553895 21130 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:27.555678 20090 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.557794 21130 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:27.558471 21130 raft_consensus.cc:697] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:27.558940 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 44081.4us
W20260317 16:18:27.560387 20236 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:27.560890 20236 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.563791 20322 tablet_bootstrap.cc:654] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:27.518854 21073 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:27.511240 20016 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.504346 20973 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.577891 21073 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.581655 20973 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.581846 21073 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:27.521358 21141 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.510092 21135 consensus_queue.cc:1059] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260317 16:18:27.522732 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:27.519497 21143 consensus_queue.cc:1059] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20260317 16:18:27.524824 21110 consensus_meta.cc:287] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.632s user 0.012s sys 0.000s
I20260317 16:18:27.585320 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:27.585592 21141 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:27.586234 20119 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:27.585875 21110 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:27.519578 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 176883us
I20260317 16:18:27.525588 20123 raft_consensus.cc:1275] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260317 16:18:27.527247 21139 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:27.494509 21131 raft_consensus.cc:697] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:27.513417 20975 raft_consensus.cc:3060] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:27.510169 21155 consensus_queue.cc:1059] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260317 16:18:27.533814 20124 raft_consensus.cc:1275] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:27.535126 20270 raft_consensus.cc:1275] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:27.541064 20321 raft_consensus.cc:359] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.541499 21124 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.494397 21153 raft_consensus.cc:2749] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:27.542255 20887 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.546986 21119 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.548516 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:27.624838 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 76431.9us
I20260317 16:18:27.549172 21157 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:27.521216 20269 raft_consensus.cc:1240] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260317 16:18:27.570315 20892 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:27.570086 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.575533 21156 raft_consensus.cc:2749] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:27.593047 20123 pending_rounds.cc:85] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Aborting all ops after (but not including) 1
W20260317 16:18:27.610136 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 80413.8us
I20260317 16:18:27.615063 20321 raft_consensus.cc:385] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 16:18:27.619313 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 108168us
I20260317 16:18:27.667171 20016 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:27.622368 21110 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.559728 21130 consensus_queue.cc:237] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:27.631330 20238 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.619676 21145 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:27.623173 21149 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:27.649942 20321 raft_consensus.cc:740] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:27.703380 20321 consensus_queue.cc:260] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.627714 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:27.635658 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:27.635718 21094 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:27.635645 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:27.760282 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:27.713085 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 81891.1us
I20260317 16:18:27.695930 20123 pending_rounds.cc:107] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260317 16:18:27.627774 20887 consensus_queue.cc:1048] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.084s
I20260317 16:18:27.694497 20917 raft_consensus.cc:3055] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Stepping down as leader of term 3
W20260317 16:18:27.757043 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:27.791378 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:27.792964 20088 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:27.708839 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:27.822218 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:27.842339 20016 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:27.779455 20893 raft_consensus.cc:2804] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:27.789544 20123 raft_consensus.cc:2889] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:27.789264 20268 raft_consensus.cc:3060] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:27.790963 20192 raft_consensus.cc:1240] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:27.779291 21165 raft_consensus.cc:2804] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:27.854099 20321 ts_tablet_manager.cc:1434] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.434s user 0.005s sys 0.003s
I20260317 16:18:27.695400 20892 consensus_queue.cc:1048] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
I20260317 16:18:27.791898 20917 raft_consensus.cc:740] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:27.796522 20973 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:27.796583 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:27.714890 21145 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.864885 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:27.847105 21100 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:27.865357 21124 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.851426 20270 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:27.796522 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:27.862510 20893 raft_consensus.cc:697] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:27.867169 21138 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.867852 21005 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:27.859683 21169 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:27.868115 20980 consensus_queue.cc:1059] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:27.870198 21134 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:27.845610 21131 consensus_queue.cc:237] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.872292 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.846901 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:27.848281 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 174) took 1119 ms. Trace:
I20260317 16:18:27.803014 20826 consensus_queue.cc:1048] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
I20260317 16:18:27.904168 21005 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.872790 21119 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.792142 20264 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:27.911103 20980 raft_consensus.cc:3055] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:27.911410 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.911202 20893 consensus_queue.cc:237] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:27.912400 20264 raft_consensus.cc:2393] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:27.913434 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:27.914050 20264 raft_consensus.cc:3060] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:27.914446 20273 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:27.861686 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.915663 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.916693 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:27.916848 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.918119 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:27.897145 20322 tablet_bootstrap.cc:492] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:27.920156 20322 ts_tablet_manager.cc:1403] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.471s user 0.019s sys 0.007s
I20260317 16:18:27.924381 21145 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:27.862097 21165 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:27.925235 20091 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.907622 21138 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:27.911657 20980 raft_consensus.cc:740] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:27.928000 20973 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:27.927904 20980 consensus_queue.cc:260] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:27.928467 20973 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:27.928589 20043 raft_consensus.cc:1275] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:27.928943 20973 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:27.929052 20980 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:27.929183 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:27.929906 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.875424 21164 raft_consensus.cc:3055] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:27.930987 21164 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:27.896806 21100 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:27.932005 21164 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:27.933220 21164 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:27.919976 20050 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:27.924965 20917 consensus_queue.cc:260] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:27.926265 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:27.904088 20195 raft_consensus.cc:2468] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:27.907382 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:27.910218 21005 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:27.926394 20091 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.918246 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:27.940630 20091 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:27.933403 21100 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:27.925112 21165 raft_consensus.cc:3060] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:27.930605 20043 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:27.919085 21037 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:27.926551 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:27.920436 20193 rpcz_store.cc:276] 0317 16:18:26.729019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.731381 (+ 2362us) service_pool.cc:225] Handling call
0317 16:18:27.619820 (+888439us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:27.844603 (+224783us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122094336}
I20260317 16:18:27.928473 20975 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
W20260317 16:18:27.987291 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 46766.9us
I20260317 16:18:27.943832 21094 raft_consensus.cc:697] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:27.986821 21124 raft_consensus.cc:2804] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:27.994644 20041 raft_consensus.cc:2410] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260317 16:18:27.992815 21133 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:27.994441 20273 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 0.
I20260317 16:18:28.013149 21094 consensus_queue.cc:237] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:28.016566 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 117616us
I20260317 16:18:28.010707 21124 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:28.001623 21005 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:28.027974 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 461820us
I20260317 16:18:27.998054 20267 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:28.011207 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:28.013914 20042 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
W20260317 16:18:28.054082 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:28.050906 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 126616us
W20260317 16:18:28.051548 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:27.996830 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 55719.7us
I20260317 16:18:28.057011 20016 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.059208 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.059005 20016 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.060181 20120 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:28.060673 20120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 173) took 1154 ms. Trace:
I20260317 16:18:28.060986 20120 rpcz_store.cc:276] 0317 16:18:26.906996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.911598 (+ 4602us) service_pool.cc:225] Handling call
0317 16:18:28.059033 (+1147435us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:28.060607 (+ 1574us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7821696}
I20260317 16:18:28.062528 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.063409 20975 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:28.063982 21169 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.064515 20016 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:28.065171 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.065878 20016 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.066058 21169 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.066884 20975 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:28.068856 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:28.070304 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.072700 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.050490 21124 raft_consensus.cc:3060] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:28.146602 20124 raft_consensus.cc:1275] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:28.147809 20321 tablet_bootstrap.cc:492] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:28.078936 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.048581 20270 raft_consensus.cc:1275] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:28.083372 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:28.084324 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:28.088027 20268 raft_consensus.cc:2468] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:28.089005 20318 tablet_bootstrap.cc:492] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:28.089300 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 135) took 1074 ms. Trace:
I20260317 16:18:28.091688 20322 raft_consensus.cc:359] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.093180 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.100574 21068 raft_consensus.cc:3055] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:28.100386 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.175484 20268 raft_consensus.cc:1240] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:28.176458 20199 raft_consensus.cc:1240] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:28.176887 20268 raft_consensus.cc:1240] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:28.177424 20270 raft_consensus.cc:1240] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:28.177729 20199 raft_consensus.cc:1275] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:28.178368 20268 raft_consensus.cc:1275] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:28.053504 21005 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.172142 20273 raft_consensus.cc:2468] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:28.103138 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:28.105310 20047 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:28.175539 21068 raft_consensus.cc:740] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:28.181034 20271 raft_consensus.cc:2393] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:28.181439 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.182380 20047 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:28.181912 21068 consensus_queue.cc:260] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:28.183267 21068 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:28.183463 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.184729 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:28.185695 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.107092 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 174) took 1199 ms. Trace:
I20260317 16:18:28.109897 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.110916 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:28.112089 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.197295 20199 raft_consensus.cc:1275] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:28.112433 20267 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:28.113344 20266 raft_consensus.cc:1275] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260317 16:18:28.114163 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 48378.9us
I20260317 16:18:28.117100 20975 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.117520 20196 raft_consensus.cc:1240] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:28.017035 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.049615 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.016083 21133 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:28.126482 20269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 113) took 1165 ms. Trace:
I20260317 16:18:28.134829 20973 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:28.130353 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.225531 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 111) took 1269 ms. Trace:
I20260317 16:18:28.226037 20266 rpcz_store.cc:276] 0317 16:18:26.956008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.957983 (+ 1975us) service_pool.cc:225] Handling call
0317 16:18:28.112712 (+1154729us) spinlock_profiling.cc:243] Waited 7.48 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:28.112942 (+ 230us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:28.206947 (+ 94005us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.225458 (+ 18511us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17204096}
I20260317 16:18:28.226985 20125 raft_consensus.cc:1275] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:28.227159 20973 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:28.228276 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:28.138804 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 139) took 1566 ms. Trace:
I20260317 16:18:28.229040 20116 rpcz_store.cc:276] 0317 16:18:26.572641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.579353 (+ 6712us) service_pool.cc:225] Handling call
0317 16:18:26.815757 (+236404us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:26.818462 (+ 2705us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:26.818675 (+ 213us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:26.993184 (+174509us) log.cc:844] Serialized 14 byte log entry
0317 16:18:28.064353 (+1071169us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b18001b1f60. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea38ea785c 00007fea3623539b 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea36251b92 00007fea38e995ab 00007fea38e9f0c3 00007fea38e9f5da
0317 16:18:28.137310 (+ 72957us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:28.137397 (+ 87us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:28.137469 (+ 72us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.138420 (+ 951us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:28.138712 (+ 292us) inbound_call.cc:173] Queueing success response
0317 16:18:28.227864 (+ 89152us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b18001b1f60. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36252383 00007fea36254751 00007fea36254709 00007fea36254699 00007fea36254661 00007fea3625395d 0000000000379254 0000000000379099 00007fea3623574a
Metrics: {"prepare.queue_time_us":189921,"prepare.run_cpu_time_us":10128,"prepare.run_wall_time_us":131535,"raft.queue_time_us":2663,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":31892992,"thread_start_us":305089,"threads_started":2,"wal-append.queue_time_us":180508}
I20260317 16:18:28.231128 21133 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:28.142246 20264 raft_consensus.cc:2468] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:28.225317 20269 rpcz_store.cc:276] 0317 16:18:26.959175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.967872 (+ 8697us) service_pool.cc:225] Handling call
0317 16:18:27.027144 (+ 59272us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:27.643144 (+616000us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.118287 (+475143us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54006842d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:28.124063 (+ 5776us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":820591360}
I20260317 16:18:28.171725 20042 rpcz_store.cc:276] 0317 16:18:27.015705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:27.157350 (+141645us) service_pool.cc:225] Handling call
0317 16:18:27.566443 (+409093us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:28.089232 (+522789us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16064384}
I20260317 16:18:28.234295 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.234925 20042 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
W20260317 16:18:28.235668 20239 leader_election.cc:343] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:28.235836 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.144057 20238 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:28.236851 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 92892.5us
I20260317 16:18:28.238382 20270 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260317 16:18:28.144596 20043 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:28.158905 20162 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.239519 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:28.169473 20318 ts_tablet_manager.cc:1403] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.723s user 0.020s sys 0.011s
I20260317 16:18:28.239873 20091 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:28.241788 21103 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.175194 20123 raft_consensus.cc:2410] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 4 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260317 16:18:28.188526 20090 leader_election.cc:343] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:28.186990 21184 consensus_queue.cc:1048] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260317 16:18:28.186913 20121 rpcz_store.cc:276] 0317 16:18:26.907289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.915397 (+ 8108us) service_pool.cc:225] Handling call
0317 16:18:28.090087 (+1174690us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:28.090281 (+ 194us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:28.090632 (+ 351us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.107011 (+ 16379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3209088}
I20260317 16:18:28.243314 21138 raft_consensus.cc:2804] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:28.243681 20121 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: []
I20260317 16:18:28.192026 20198 raft_consensus.cc:2393] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 3: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:28.196687 20122 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:28.253175 20318 raft_consensus.cc:359] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:28.254232 20318 raft_consensus.cc:385] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:28.254638 20318 raft_consensus.cc:740] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:28.243448 20123 raft_consensus.cc:1275] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:28.202072 20016 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.192691 20893 consensus_queue.cc:1048] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260317 16:18:28.243919 20924 raft_consensus.cc:2804] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:28.213938 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:28.192907 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.215826 20975 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:28.221423 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:28.180576 21139 raft_consensus.cc:3055] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:28.228417 20125 raft_consensus.cc:1275] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:28.137676 21116 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:28.239676 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 80933.1us
I20260317 16:18:28.171835 20322 raft_consensus.cc:385] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:28.193795 21182 consensus_queue.cc:1048] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
W20260317 16:18:28.242707 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 54220.4us
W20260317 16:18:28.245936 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.247567 20238 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:28.250177 20321 tablet_bootstrap.cc:654] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:28.255618 20318 consensus_queue.cc:260] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:28.244061 21130 consensus_queue.cc:1048] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260317 16:18:28.244926 21154 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.212922 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.244134 21131 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:28.201999 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:28.243894 21138 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
W20260317 16:18:28.260844 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 58917.4us
I20260317 16:18:28.246620 20980 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.252281 20973 consensus_queue.cc:1059] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20260317 16:18:28.368889 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.315624 20091 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:28.263075 20119 consensus_meta.cc:287] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.676s user 0.012s sys 0.000s
I20260317 16:18:28.285348 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:28.390453 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 148726us
I20260317 16:18:28.313858 20320 tablet_bootstrap.cc:492] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:28.261355 21005 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:28.428093 20016 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:28.276010 20236 leader_election.cc:336] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.557s (SENT)
I20260317 16:18:28.433609 20236 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:28.420426 20322 raft_consensus.cc:740] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:28.435854 20165 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.260655 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:28.444813 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.441092 20318 ts_tablet_manager.cc:1434] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.201s user 0.011s sys 0.004s
I20260317 16:18:28.264354 21037 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.444967 20162 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:28.446859 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 199402us
I20260317 16:18:28.440611 20322 consensus_queue.cc:260] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.285048 20121 raft_consensus.cc:1240] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:28.319319 20917 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:28.286806 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.285238 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:28.269117 21139 raft_consensus.cc:740] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:28.400048 21006 raft_consensus.cc:2804] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:28.276154 20047 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:28.395226 20041 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:28.483835 20318 tablet_bootstrap.cc:492] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:28.421346 21138 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:28.498229 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 222237us
I20260317 16:18:28.301308 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:28.309391 21135 raft_consensus.cc:3055] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:28.470191 20162 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:28.273725 20924 raft_consensus.cc:697] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:28.280539 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:28.315380 20044 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:28.305127 20967 raft_consensus.cc:3055] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:28.257795 21124 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.468886 20016 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:28.271222 20975 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:28.312906 20191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 178) took 1565 ms. Trace:
I20260317 16:18:28.296311 20042 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:28.487803 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:28.488166 21005 raft_consensus.cc:3055] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:28.439067 21157 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260317 16:18:28.488547 20892 consensus_queue.cc:1048] T aa7ec7cb7aad43e8950899557d4022f7 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:28.448460 21115 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:28.489329 20048 raft_consensus.cc:1275] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:28.447237 21175 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:28.465054 21068 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:28.459174 21167 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.465484 20264 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:28.491328 20122 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:28.468361 21156 consensus_queue.cc:1048] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260317 16:18:28.447424 20980 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.472379 21036 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.464151 21184 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.489372 19945 catalog_manager.cc:5671] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 2 to 3, leader changed from d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196) to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 3 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:28.478261 21006 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:28.445451 21002 consensus_queue.cc:1059] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260317 16:18:28.478720 21169 consensus_queue.cc:1048] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260317 16:18:28.504649 21135 raft_consensus.cc:740] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:28.435868 20119 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:28.508713 21005 raft_consensus.cc:740] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:28.511291 20236 leader_election.cc:336] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.906s (SENT)
I20260317 16:18:28.430974 21154 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:28.505215 20195 raft_consensus.cc:2393] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:28.498051 21182 consensus_queue.cc:1048] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
W20260317 16:18:28.486387 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.514096 20322 ts_tablet_manager.cc:1434] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.594s user 0.009s sys 0.008s
W20260317 16:18:28.548694 20322 ts_tablet_manager.cc:1467] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: Tablet startup took 628ms
I20260317 16:18:28.504999 20967 raft_consensus.cc:740] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:28.506218 20162 leader_election.cc:343] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:28.510030 20826 raft_consensus.cc:2749] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:28.509850 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:28.518906 21072 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.540539 20236 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:28.531340 20122 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 176) took 1619 ms (client timeout 1770 ms). Trace:
I20260317 16:18:28.542135 20120 raft_consensus.cc:2393] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:28.497570 21165 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.537236 19945 catalog_manager.cc:5671] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } }
I20260317 16:18:28.531992 21006 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:28.502910 21153 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.555140 20162 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:28.549397 20322 ts_tablet_manager.cc:1469] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321: Trace:
0317 16:18:05.305348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:05.305512 (+ 164us) service_pool.cc:225] Handling call
0317 16:18:05.309226 (+ 3714us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:05.309311 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.359393 (+ 50082us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.388557 (+ 29164us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.389045 (+ 488us) inbound_call.cc:173] Queueing success response
0317 16:18:27.432697 (+22043652us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:27.920510 (+487813us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:28.089726 (+169216us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:28.090231 (+ 505us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":32053,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"tablet-open.queue_time_us":22043732}
W20260317 16:18:28.543999 21164 consensus_meta.cc:287] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.610s user 0.012s sys 0.001s
I20260317 16:18:28.536208 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 141) took 1341 ms. Trace:
I20260317 16:18:28.524031 20938 consensus_queue.cc:1048] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260317 16:18:28.543239 20047 raft_consensus.cc:1240] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260317 16:18:28.518949 21115 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.518868 21158 consensus_queue.cc:1059] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.291s
I20260317 16:18:28.615998 20124 raft_consensus.cc:1240] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260317 16:18:28.619823 21191 log.cc:927] Time spent T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260317 16:18:28.726361 20050 raft_consensus.cc:2393] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:28.512071 20191 rpcz_store.cc:276] 0317 16:18:26.747773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.752023 (+ 4250us) service_pool.cc:225] Handling call
0317 16:18:27.008692 (+256669us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:28.074644 (+1065952us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:28.312166 (+237522us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:28.312343 (+ 177us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:28.312536 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:28.312575 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:28.312617 (+ 42us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:28.312654 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.312716 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:28.312840 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227379456}
I20260317 16:18:28.627490 20195 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:28.627221 20967 consensus_queue.cc:260] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:28.628262 20924 consensus_queue.cc:237] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.520565 21139 consensus_queue.cc:260] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.631943 21124 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:28.637020 20044 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 142) took 1427 ms (client timeout 1766 ms). Trace:
W20260317 16:18:28.637475 21178 log.cc:927] Time spent T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.374s user 0.000s sys 0.001s
W20260317 16:18:28.757346 20044 rpcz_store.cc:269] 0317 16:18:27.209185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:27.210662 (+ 1477us) service_pool.cc:225] Handling call
0317 16:18:28.636782 (+1426120us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54002440d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:28.636944 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2217170688}
I20260317 16:18:28.642369 21036 raft_consensus.cc:3055] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:28.759584 21036 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:28.759824 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 149) took 1239 ms. Trace:
I20260317 16:18:28.760423 20124 rpcz_store.cc:276] 0317 16:18:27.520852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.147482 (+626630us) service_pool.cc:225] Handling call
0317 16:18:28.542934 (+395452us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:28.729995 (+187061us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.759381 (+ 29386us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54000acb50. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:28.759746 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":897471360}
I20260317 16:18:28.760890 21036 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:28.645169 21190 log.cc:927] Time spent T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260317 16:18:28.551117 20119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 184) took 1029 ms. Trace:
I20260317 16:18:28.762457 20264 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:28.763375 20119 rpcz_store.cc:276] 0317 16:18:27.518262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:27.584908 (+ 66646us) service_pool.cc:225] Handling call
0317 16:18:28.547217 (+962309us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:28.658743 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.531674 21037 raft_consensus.cc:2804] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:28.765251 21037 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:28.765992 21037 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:28.766873 20967 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:28.732123 20269 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:28.745589 20190 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:28.771754 20046 raft_consensus.cc:2410] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260317 16:18:28.775094 21133 consensus_meta.cc:287] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.543s user 0.008s sys 0.005s
I20260317 16:18:28.754580 20320 tablet_bootstrap.cc:654] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:28.544653 21118 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260317 16:18:28.617982 20973 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.657612 21141 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.661890 20980 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:28.666775 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 96108.4us
W20260317 16:18:28.691697 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 222942us
I20260317 16:18:28.799414 20016 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:28.690619 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 179407us
W20260317 16:18:28.703006 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 196809us
I20260317 16:18:28.689846 21072 raft_consensus.cc:2749] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:28.713922 20045 rpcz_store.cc:276] 0317 16:18:27.191824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:27.202468 (+ 10644us) service_pool.cc:225] Handling call
0317 16:18:28.301078 (+1098610us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:28.533604 (+232526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8196352}
I20260317 16:18:28.687227 21135 consensus_queue.cc:260] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:28.735616 21186 log.cc:927] Time spent T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.266s user 0.001s sys 0.000s
I20260317 16:18:28.737341 21115 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:28.732412 21080 raft_consensus.cc:3055] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Stepping down as leader of term 3
W20260317 16:18:28.689999 20122 rpcz_store.cc:269] 0317 16:18:26.907996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:26.911295 (+ 3299us) service_pool.cc:225] Handling call
0317 16:18:28.110679 (+1199384us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:28.526954 (+416275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15044992}
I20260317 16:18:28.741490 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 186) took 1172 ms. Trace:
I20260317 16:18:28.742553 21154 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:28.673918 21005 consensus_queue.cc:260] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.718401 20117 raft_consensus.cc:1275] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260317 16:18:28.832151 21002 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.746486 20322 tablet_bootstrap.cc:492] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:28.755968 21165 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:28.821151 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:28.820713 20088 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:28.902750 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 82100.7us
I20260317 16:18:28.903656 20088 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:28.905753 20088 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.776147 21133 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.727648 19945 catalog_manager.cc:5671] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 1 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
W20260317 16:18:28.824179 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 312793us
I20260317 16:18:28.792361 21036 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
W20260317 16:18:28.911882 20238 leader_election.cc:343] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:28.912535 20238 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:28.834751 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:28.841749 20016 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.833354 20091 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:28.915771 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 82508.1us
I20260317 16:18:28.839833 20935 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.842892 20050 raft_consensus.cc:1275] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:28.846510 20162 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:28.852557 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.829397 21164 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:28.840955 21156 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.916831 20091 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:28.810119 21141 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:28.720988 20826 raft_consensus.cc:2749] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:28.831877 21131 consensus_queue.cc:1059] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260317 16:18:28.918100 21158 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:28.850991 21169 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.843842 21080 raft_consensus.cc:740] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:28.798070 20973 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.924553 20893 consensus_queue.cc:1048] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20260317 16:18:28.976325 20893 raft_consensus.cc:493] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.976650 20973 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:28.930389 20236 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.962s (SENT) [suppressed 33 similar messages]
I20260317 16:18:28.929077 20892 raft_consensus.cc:2749] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:28.934108 20050 raft_consensus.cc:1275] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260317 16:18:28.914724 21193 log.cc:927] Time spent T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.198s user 0.002s sys 0.000s
I20260317 16:18:28.804031 21139 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:28.851787 21135 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:28.905295 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:28.941836 20016 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:28.900210 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 170) took 1008 ms. Trace:
I20260317 16:18:28.926873 20272 raft_consensus.cc:1240] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: []
W20260317 16:18:28.988108 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 46497.2us
I20260317 16:18:28.949894 20239 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:28.924764 21127 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:28.939436 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:28.958178 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.955188 20118 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:28.958662 21130 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:28.959033 20980 raft_consensus.cc:2804] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:28.961169 21006 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.942028 20120 raft_consensus.cc:1275] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:28.962472 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:28.906235 21002 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:28.926740 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:28.946853 21165 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:28.963423 20826 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:28.964849 20318 tablet_bootstrap.cc:654] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:28.967483 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:28.928867 21116 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:28.979636 19945 catalog_manager.cc:5671] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 1 to 2, leader changed from 9a3afb098d6a45f0b405566771a1157b (127.16.123.193) to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 2 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } }
W20260317 16:18:28.985095 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 54756.4us
I20260317 16:18:28.984804 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:28.991719 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 83582us
I20260317 16:18:29.055469 20236 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:28.955483 20047 raft_consensus.cc:1275] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:28.976852 20893 raft_consensus.cc:515] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:29.033622 20980 raft_consensus.cc:697] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:28.931677 20935 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:29.007131 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:28.974095 21080 consensus_queue.cc:260] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:28.941547 21156 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:29.022503 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:29.025035 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 178653us
I20260317 16:18:28.989897 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:29.000697 20892 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:28.995606 21131 raft_consensus.cc:2749] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:28.998203 21127 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:29.045902 20190 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:29.046164 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:28.979447 21169 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.056819 20236 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:29.017201 20122 raft_consensus.cc:2376] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5. Re-sending same reply.
I20260317 16:18:28.988973 20016 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:29.060729 19945 catalog_manager.cc:5671] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: UNKNOWN } } }
W20260317 16:18:29.100584 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 111707us
W20260317 16:18:29.035873 20917 consensus_meta.cc:287] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.560s user 0.011s sys 0.006s
I20260317 16:18:29.063822 20088 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:29.069903 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 144549us
I20260317 16:18:29.057771 20268 raft_consensus.cc:2393] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:29.023149 21130 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:28.988106 20041 rpcz_store.cc:276] 0317 16:18:27.885623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.502438 (+616815us) service_pool.cc:225] Handling call
0317 16:18:28.527292 (+ 24854us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:28.555583 (+ 28291us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:28.659900 (+104317us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:28.667428 (+ 7528us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:28.670699 (+ 3271us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.838222 (+167523us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5400566654. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:28.842435 (+ 4213us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:28.894173 (+ 51738us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25947904}
W20260317 16:18:29.084087 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:29.067862 21194 log.cc:927] Time spent T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.217s user 0.002s sys 0.000s
I20260317 16:18:29.062561 21145 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:29.059127 20046 raft_consensus.cc:2376] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2. Re-sending same reply.
I20260317 16:18:29.080505 21127 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:29.072353 21036 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:29.140110 21036 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.116524 21002 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.082662 21154 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.083990 20892 raft_consensus.cc:697] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:28.898401 20116 rpcz_store.cc:276] 0317 16:18:27.568659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.231135 (+662476us) service_pool.cc:225] Handling call
0317 16:18:28.231548 (+ 413us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:28.236919 (+ 5371us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:28.236953 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:28.237474 (+ 521us) log.cc:844] Serialized 26 byte log entry
0317 16:18:28.263432 (+ 25958us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:28.465652 (+202220us) log.cc:844] Serialized 14 byte log entry
0317 16:18:28.610195 (+144543us) spinlock_profiling.cc:243] Waited 6.91 ms on lock 0x7b18003b9a40. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea361c74ca 00007fea3f8595c9 00007fea360871c0 00007fea3624f474 00007fea3624f3e5 00007fea392f7f90 00007fea3930a646 00007fea3931b1b7 00007fea3931b138
0317 16:18:28.610508 (+ 313us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.610624 (+ 116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:28.741035 (+130411us) raft_consensus.cc:1685] finished
0317 16:18:28.741187 (+ 152us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:28.741392 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46820,"prepare.queue_time_us":98921,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":54,"raft.queue_time_us":146326,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":18073,"spinlock_wait_cycles":15892224,"thread_start_us":21019,"threads_started":1,"wal-append.queue_time_us":271747}
I20260317 16:18:29.146502 20892 consensus_queue.cc:237] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:29.147464 20199 raft_consensus.cc:1275] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:29.148962 20935 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.072554 21116 raft_consensus.cc:697] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:29.153386 21116 consensus_queue.cc:237] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:29.145150 21154 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.120338 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.080894 21119 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.050629 21165 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.066064 20826 raft_consensus.cc:697] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
W20260317 16:18:29.159299 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 95562.6us
W20260317 16:18:29.159232 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 102538us
I20260317 16:18:29.158327 20239 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:29.159754 20091 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:29.122615 20322 tablet_bootstrap.cc:654] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:29.164387 20091 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:29.160382 20236 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:29.118640 20321 tablet_bootstrap.cc:492] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:29.117304 19945 catalog_manager.cc:5671] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:29.171928 20321 ts_tablet_manager.cc:1403] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.025s user 0.027s sys 0.028s
I20260317 16:18:29.173130 20049 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:29.184147 19945 catalog_manager.cc:5671] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 1 to 2, leader changed from d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196) to 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194). New cstate: current_term: 2 leader_uuid: "795c49e9595f45bdb8d6b568498abf1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:29.156313 21119 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:29.062278 21068 consensus_queue.cc:1048] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260317 16:18:29.099913 20117 raft_consensus.cc:2393] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:29.183125 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 72432.3us
I20260317 16:18:29.256199 19945 catalog_manager.cc:5671] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f reported cstate change: term changed from 1 to 2, leader changed from 9a3afb098d6a45f0b405566771a1157b (127.16.123.193) to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } }
I20260317 16:18:29.119143 20269 raft_consensus.cc:2393] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:29.078068 21141 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:29.178889 21145 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:29.156806 21154 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:29.089721 20973 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:29.189750 21202 log.cc:927] Time spent T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.143s user 0.000s sys 0.002s
I20260317 16:18:29.121668 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:29.119474 21133 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.189090 21036 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.185366 21124 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:29.122392 21080 raft_consensus.cc:3060] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:29.123075 21196 raft_consensus.cc:2804] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 16:18:29.288838 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 215850us
I20260317 16:18:29.289960 21211 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:29.188922 20924 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:29.275445 21145 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:29.275476 21169 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:29.313084 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 211505us
I20260317 16:18:29.258919 21115 consensus_queue.cc:1048] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.042s
W20260317 16:18:29.279044 20967 consensus_meta.cc:287] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.512s user 0.012s sys 0.003s
I20260317 16:18:29.122920 21131 consensus_queue.cc:1048] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.158s
I20260317 16:18:29.080657 21201 consensus_queue.cc:1048] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.198s
I20260317 16:18:29.112380 21197 consensus_queue.cc:1059] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.174s
I20260317 16:18:29.271905 21072 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:29.346271 20320 tablet_bootstrap.cc:492] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:29.347064 20320 ts_tablet_manager.cc:1403] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 1.033s user 0.025s sys 0.026s
I20260317 16:18:29.292860 21141 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260317 16:18:29.292704 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 176300us
W20260317 16:18:29.322973 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 164714us
I20260317 16:18:29.332958 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:29.302192 21196 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:29.321488 21138 consensus_meta.cc:287] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.812s user 0.009s sys 0.006s
I20260317 16:18:29.283461 20973 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.323030 21220 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.324990 21006 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.373418 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:29.329809 21169 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.344419 21068 raft_consensus.cc:2749] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:29.372938 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:29.353430 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:29.288009 20980 consensus_queue.cc:237] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:29.398808 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 238453us
I20260317 16:18:29.429163 20320 raft_consensus.cc:359] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.430032 20320 raft_consensus.cc:385] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:29.430327 20320 raft_consensus.cc:740] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:29.430122 20236 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:29.342226 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.338610 20998 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:29.380162 21215 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.431165 20318 tablet_bootstrap.cc:492] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:29.350368 21141 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:29.431173 20320 consensus_queue.cc:260] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.353899 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:29.436118 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 120548us
I20260317 16:18:29.374748 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.439154 20318 ts_tablet_manager.cc:1403] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.956s user 0.035s sys 0.007s
I20260317 16:18:29.321000 21211 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:29.375058 21118 consensus_queue.cc:1048] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260317 16:18:29.396219 21196 raft_consensus.cc:3060] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:29.324021 20826 consensus_queue.cc:237] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.478919 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.448035 20016 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:29.434405 20264 raft_consensus.cc:3060] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:29.386745 20967 consensus_queue.cc:1059] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260317 16:18:29.463140 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:29.448035 21205 log.cc:927] Time spent T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.370s user 0.000s sys 0.001s
I20260317 16:18:29.485181 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.445034 21141 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:29.395946 21138 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:29.457222 21222 raft_consensus.cc:2804] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:29.492929 21165 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.493927 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.444962 21215 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.495498 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:29.456626 20998 raft_consensus.cc:697] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:29.438428 20893 leader_election.cc:290] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:29.440687 21197 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.500320 20320 ts_tablet_manager.cc:1434] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.153s user 0.009s sys 0.000s
I20260317 16:18:29.464026 20194 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
W20260317 16:18:29.483596 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 114099us
I20260317 16:18:29.412365 21169 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.518349 21138 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:29.528270 20321 raft_consensus.cc:359] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:29.512595 21037 consensus_meta.cc:287] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.746s user 0.015s sys 0.003s
W20260317 16:18:29.536159 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 88617.6us
I20260317 16:18:29.499006 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 221) took 1597 ms. Trace:
I20260317 16:18:29.416384 21072 consensus_queue.cc:1048] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260317 16:18:29.499302 21130 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.485664 21110 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:29.545992 21204 log.cc:927] Time spent T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.361s user 0.000s sys 0.002s
I20260317 16:18:29.387202 21220 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.459079 21131 consensus_queue.cc:1048] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.063s
I20260317 16:18:29.483599 20924 raft_consensus.cc:697] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
F20260317 16:18:29.520166 16879 test_workload.cc:382] Timed out: Timed out waiting for Table Creation
I20260317 16:18:29.508440 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:29.780442 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:29.779670 21145 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:29.552537 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:29.782440 20272 raft_consensus.cc:1240] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
*** Check failure stack trace: ***
I20260317 16:18:29.521459 21119 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.610282 20318 raft_consensus.cc:359] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:29.783026 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 47685.4us
I20260317 16:18:29.784052 20318 raft_consensus.cc:385] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:29.537403 21165 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:29.512176 21201 consensus_queue.cc:1048] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260317 16:18:29.784714 20318 raft_consensus.cc:740] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:29.785956 20318 consensus_queue.cc:260] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
*** Aborted at 1773764309 (unix time) try "date -d @1773764309" if you are using GNU date ***
W20260317 16:18:29.615871 21158 consensus_meta.cc:287] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.660s user 0.012s sys 0.000s
I20260317 16:18:29.746167 21196 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:29.788292 21158 raft_consensus.cc:2749] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:29.617466 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 83412.1us
I20260317 16:18:29.619102 20321 raft_consensus.cc:385] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:29.791167 20321 raft_consensus.cc:740] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:29.792692 21196 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:29.626991 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:29.546711 21211 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.636250 20044 raft_consensus.cc:1275] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:29.638152 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.546831 21215 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:29.624394 21135 consensus_meta.cc:287] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.636s user 0.006s sys 0.007s
I20260317 16:18:29.625015 20271 raft_consensus.cc:2393] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
W20260317 16:18:29.834313 21139 consensus_meta.cc:287] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.849s user 0.008s sys 0.017s
I20260317 16:18:29.656430 20090 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:29.560037 20048 raft_consensus.cc:2376] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3. Re-sending same reply.
I20260317 16:18:29.546964 21197 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.652645 21169 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.629379 21037 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:29.693166 20162 leader_election.cc:336] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.752s (SENT)
I20260317 16:18:29.669534 21130 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:29.693353 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:29.862601 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:29.638753 21110 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.685988 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.708801 20196 raft_consensus.cc:1275] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260317 16:18:29.711822 21226 log.cc:927] Time spent T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260317 16:18:29.712177 20268 raft_consensus.cc:2393] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:29.680840 20973 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:29.694432 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.885653 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:29.887151 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.714632 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:29.879231 21138 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.723621 20924 consensus_queue.cc:237] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.684371 21224 consensus_queue.cc:1048] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260317 16:18:29.729148 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.734287 21220 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.746618 20016 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:29.722489 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:29.891705 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:29.892386 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 145968us
I20260317 16:18:29.894627 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:29.895277 21169 raft_consensus.cc:2749] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:29.725437 21235 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.691999 20998 consensus_queue.cc:237] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:29.896303 21235 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.758605 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.899149 20119 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:29.758921 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.899493 21216 raft_consensus.cc:3055] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:29.900045 21216 raft_consensus.cc:740] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:29.899349 20197 raft_consensus.cc:2410] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:29.900766 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:29.749475 20980 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:29.900929 21216 consensus_queue.cc:260] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.914855 21169 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:29.915459 21216 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:29.765156 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:29.766865 20269 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260317 16:18:29.767700 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:29.750824 21072 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:29.770275 21231 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:29.771552 21233 log.cc:927] Time spent T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260317 16:18:29.772459 21136 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:29.604404 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 255176us
I20260317 16:18:29.613175 21127 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:29.527107 21156 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.779985 20264 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:29.793803 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:29.805513 20318 ts_tablet_manager.cc:1434] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.358s user 0.009s sys 0.000s
I20260317 16:18:29.825579 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:29.792186 20321 consensus_queue.cc:260] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.836728 20050 raft_consensus.cc:2376] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5. Re-sending same reply.
I20260317 16:18:29.606325 21222 raft_consensus.cc:697] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:29.858909 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 202854us
W20260317 16:18:29.860677 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 167555us
W20260317 16:18:29.837442 21230 log.cc:927] Time spent T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.148s user 0.000s sys 0.007s
I20260317 16:18:29.832868 20892 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.865609 20239 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:29.952351 20125 raft_consensus.cc:1240] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:29.918828 20047 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
W20260317 16:18:29.952751 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 87255.6us
I20260317 16:18:29.865803 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.866396 20270 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:29.868039 21165 raft_consensus.cc:2749] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:29.869190 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:29.869345 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 199) took 1037 ms. Trace:
I20260317 16:18:29.871587 21037 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:29.876982 21197 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:29.877676 21130 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.884452 20273 raft_consensus.cc:1240] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260317 16:18:29.900128 20041 raft_consensus.cc:2393] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:29.761745 20322 tablet_bootstrap.cc:492] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:29.986351 20238 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:29.988106 20322 ts_tablet_manager.cc:1403] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 1.242s user 0.019s sys 0.022s
I20260317 16:18:29.860023 21211 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 16:18:29.986562 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:29.623919 20193 rpcz_store.cc:276] 0317 16:18:27.888004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.151538 (+263534us) service_pool.cc:225] Handling call
0317 16:18:28.151957 (+ 419us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:28.189156 (+ 37199us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:28.193821 (+ 4665us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:28.324055 (+130234us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b2c0024e3a8. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea37bfe9b1 00007fea37c0dfe2 00007fea3933bb01 00007fea3930036a 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22
0317 16:18:28.324502 (+ 447us) log.cc:844] Serialized 26 byte log entry
0317 16:18:28.671811 (+347309us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:28.674367 (+ 2556us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:28.680642 (+ 6275us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:29.432232 (+751590us) raft_consensus.cc:1685] finished
0317 16:18:29.433884 (+ 1652us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:29.485402 (+ 51518us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2294,"raft.queue_time_us":3433,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":1728,"spinlock_wait_cycles":11839872,"thread_start_us":333094,"threads_started":1,"wal-append.queue_time_us":473128}
I20260317 16:18:29.991107 21207 consensus_queue.cc:1048] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 16:18:29.879542 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 127) took 1063 ms. Trace:
I20260317 16:18:29.915493 21169 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:29.992281 20271 rpcz_store.cc:276] 0317 16:18:28.800753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.809822 (+ 9069us) service_pool.cc:225] Handling call
0317 16:18:29.495374 (+685552us) spinlock_profiling.cc:243] Waited 8.81 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:29.864220 (+368846us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20252416}
I20260317 16:18:29.993690 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:29.883738 21005 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:29.889195 21138 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.994947 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.996495 21138 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:29.996726 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:29.901783 20046 raft_consensus.cc:1240] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:29.888278 20194 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:29.941648 20199 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:29.920325 20196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 191) took 1109 ms. Trace:
I20260317 16:18:29.953889 20090 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:29.925851 20267 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:29.949173 20043 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:29.955216 20091 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:29.956053 20239 leader_election.cc:343] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:29.939705 21006 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:30.015086 21245 log.cc:927] Time spent T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20260317 16:18:29.940209 21136 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:29.930648 21199 consensus_queue.cc:1048] T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20260317 16:18:30.015385 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 160) took 1211 ms. Trace:
I20260317 16:18:29.927637 21072 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:29.978314 20318 tablet_bootstrap.cc:492] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:29.977641 20321 ts_tablet_manager.cc:1434] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.805s user 0.002s sys 0.025s
W20260317 16:18:30.018384 20321 ts_tablet_manager.cc:1467] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: Tablet startup took 845ms
I20260317 16:18:29.983768 21145 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:29.945210 21215 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:29.986624 20045 raft_consensus.cc:1275] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:29.977736 21157 consensus_queue.cc:1048] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260317 16:18:29.915072 20980 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.003613 20236 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:29.967778 20117 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260317 16:18:30.008956 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 55145.9us
I20260317 16:18:29.956964 21139 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:29.955212 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:30.061023 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.011003 20123 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 195) took 1180 ms. Trace:
I20260317 16:18:30.011469 21124 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:29.962392 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:30.010429 20238 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:30.014230 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 58262.1us
I20260317 16:18:30.014847 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:30.017051 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 61911.1us
I20260317 16:18:30.073781 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.016296 21165 consensus_queue.cc:1059] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260317 16:18:30.024012 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:29.956506 20924 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:30.016855 20162 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:30.075848 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 59096.6us
I20260317 16:18:29.982434 20165 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:29.951987 20826 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:29.930989 21231 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.989579 21220 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:29.967587 20935 consensus_queue.cc:1048] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260317 16:18:30.085351 20826 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:29.994223 21242 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.086555 21242 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:29.943852 21135 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.081447 20047 raft_consensus.cc:2376] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5. Re-sending same reply.
I20260317 16:18:30.088012 20826 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.087874 21135 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.088651 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:29.993705 21169 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.089659 21036 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:29.981663 20116 rpcz_store.cc:276] 0317 16:18:28.831495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.146468 (+314973us) service_pool.cc:225] Handling call
0317 16:18:29.264533 (+118065us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:29.446847 (+182314us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:29.446891 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:29.516760 (+ 69869us) log.cc:844] Serialized 26 byte log entry
0317 16:18:29.723139 (+206379us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:29.861574 (+138435us) log.cc:844] Serialized 14 byte log entry
0317 16:18:29.868881 (+ 7307us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:29.868997 (+ 116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:29.869073 (+ 76us) raft_consensus.cc:1685] finished
0317 16:18:29.869090 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:29.869271 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3886,"prepare.queue_time_us":154871,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":59763,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":26937,"spinlock_wait_cycles":125184,"thread_start_us":312929,"threads_started":3,"wal-append.queue_time_us":159762}
I20260317 16:18:30.051680 20016 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:30.045521 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.091279 20044 raft_consensus.cc:2393] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:30.091370 21164 raft_consensus.cc:3055] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:30.091955 21164 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:30.092381 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.092770 21130 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.092909 20016 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:30.053397 21127 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.093770 21169 raft_consensus.cc:2749] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:29.990092 21154 consensus_queue.cc:1048] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260317 16:18:30.094309 20967 raft_consensus.cc:3055] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:30.094395 21169 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:30.054466 20980 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.094807 20967 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:30.018784 20321 ts_tablet_manager.cc:1469] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b: Trace:
0317 16:18:04.718201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.718380 (+ 179us) service_pool.cc:225] Handling call
0317 16:18:04.720516 (+ 2136us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.720616 (+ 100us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:04.966034 (+245418us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.251988 (+285954us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.252360 (+ 372us) inbound_call.cc:173] Queueing success response
0317 16:18:28.147247 (+22894887us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:29.173083 (+1025836us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:29.448923 (+275840us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:29.464557 (+ 15634us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":2688,"raft.queue_time_us":13552,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":15296,"tablet-open.queue_time_us":22894893}
I20260317 16:18:30.095048 21169 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.096058 20321 tablet_bootstrap.cc:492] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:30.095738 20967 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.096915 20967 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:30.097187 21169 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.097754 21169 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:30.098070 21169 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.098379 21169 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:30.098281 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:30.056413 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 52867.6us
I20260317 16:18:30.056214 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.099155 20198 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
PC: @ 0x0 (unknown)
W20260317 16:18:30.100090 20236 leader_election.cc:343] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:30.058280 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:30.059355 21072 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.059760 20041 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:29.957283 20892 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.024927 20194 raft_consensus.cc:1275] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260317 16:18:30.115470 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.071008 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.074525 20050 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:30.015440 21006 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:30.079128 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 96750.4us
I20260317 16:18:30.047804 21235 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:30.069410 21141 consensus_meta.cc:287] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.548s user 0.010s sys 0.006s
W20260317 16:18:30.081267 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 70895.8us
I20260317 16:18:30.061439 21199 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:30.064333 21196 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.061339 21037 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.084431 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.085002 21236 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.090150 21036 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.189328 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:30.006594 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:30.057080 20090 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:30.195164 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 138166us
I20260317 16:18:30.196131 20090 leader_election.cc:304] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:30.100538 20236 leader_election.cc:304] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:30.199520 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 99452us
I20260317 16:18:30.099489 21136 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.070959 21197 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:30.106640 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.090360 21135 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.192947 21216 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.024951 21201 raft_consensus.cc:2804] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:30.158438 21141 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.207252 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.207561 21124 raft_consensus.cc:2749] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:30.137147 20322 raft_consensus.cc:359] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.015908 20124 rpcz_store.cc:276] 0317 16:18:28.803681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.810497 (+ 6816us) service_pool.cc:225] Handling call
0317 16:18:28.925214 (+114717us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:29.023191 (+ 97977us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:29.026113 (+ 2922us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:29.045101 (+ 18988us) log.cc:844] Serialized 26 byte log entry
0317 16:18:29.318101 (+273000us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:29.328159 (+ 10058us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:30.014973 (+686814us) spinlock_profiling.cc:243] Waited 46.8 ms on lock 0x7b5400110dd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:30.015029 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:30.015082 (+ 53us) raft_consensus.cc:1685] finished
0317 16:18:30.015102 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:30.015325 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8938,"raft.queue_time_us":37082,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":13206,"spinlock_wait_cycles":107723136,"thread_start_us":256314,"threads_started":1,"wal-append.queue_time_us":276792}
I20260317 16:18:30.208335 20322 raft_consensus.cc:385] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:30.208559 21141 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.208767 20322 raft_consensus.cc:740] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:30.005887 20196 rpcz_store.cc:276] 0317 16:18:28.808271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.809989 (+ 1718us) service_pool.cc:225] Handling call
0317 16:18:29.694611 (+884622us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:29.694895 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:29.879407 (+184512us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:29.917900 (+ 38493us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6949120}
I20260317 16:18:30.155810 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.137444 20267 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:30.107568 21220 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.109639 21139 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:30.063650 21222 consensus_queue.cc:237] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.136415 20194 pending_rounds.cc:85] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting all ops after (but not including) 2
I20260317 16:18:30.224783 20238 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:30.194867 21036 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:30.183396 21165 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.207283 20973 raft_consensus.cc:2804] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:30.207103 21216 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.082197 21241 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:30.062476 21215 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.225430 21201 raft_consensus.cc:697] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:30.230031 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:30.225265 20322 consensus_queue.cc:260] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.206054 21235 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:30.227720 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.092782 21164 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.158535 20893 consensus_queue.cc:1059] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.066s
I20260317 16:18:30.180064 21231 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.236475 20973 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.187541 20165 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:30.231835 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 138996us
I20260317 16:18:30.081246 20924 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.215034 21145 raft_consensus.cc:2749] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:30.236905 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.230253 21220 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.223836 21197 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.225678 21135 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.327301 21135 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.328429 21197 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:30.329164 21072 raft_consensus.cc:3055] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:30.329556 21072 raft_consensus.cc:740] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:30.231451 20194 pending_rounds.cc:107] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260317 16:18:30.330338 20194 raft_consensus.cc:2889] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
*** SIGABRT (@0x3e8000041ef) received by PID 16879 (TID 0x7fea31e44840) from PID 16879; stack trace: ***
I20260317 16:18:30.330554 21072 consensus_queue.cc:260] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.332108 21072 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:30.332336 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.223860 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 155) took 1411 ms. Trace:
I20260317 16:18:30.334560 20042 rpcz_store.cc:276] 0317 16:18:28.812488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.830696 (+ 18208us) service_pool.cc:225] Handling call
0317 16:18:30.073865 (+1243169us) spinlock_profiling.cc:243] Waited 8.52 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:30.100169 (+ 26304us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.139875 (+ 39706us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:30.155754 (+ 15879us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:30.160119 (+ 4365us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:30.180767 (+ 20648us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:30.191130 (+ 10363us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:30.212762 (+ 21632us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19605760}
I20260317 16:18:30.219794 20318 tablet_bootstrap.cc:654] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:30.336406 20047 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:30.192829 21236 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.225108 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.339126 21236 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:30.063091 20123 rpcz_store.cc:276] 0317 16:18:28.830379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.097848 (+267469us) service_pool.cc:225] Handling call
0317 16:18:29.124004 (+ 26156us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:29.305928 (+181924us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:29.314872 (+ 8944us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:29.402848 (+ 87976us) log.cc:844] Serialized 26 byte log entry
0317 16:18:29.534950 (+132102us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:29.535044 (+ 94us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:29.762949 (+227905us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b540016dd54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:29.763017 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:29.967288 (+204271us) raft_consensus.cc:1685] finished
0317 16:18:29.967306 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:30.010939 (+ 43633us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30843,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6472,"spinlock_wait_cycles":326619904,"thread_start_us":99913,"threads_started":1,"wal-append.queue_time_us":247633}
I20260317 16:18:30.340834 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.341610 20050 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:30.264098 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:30.228928 21139 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.234884 21156 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:30.244904 20238 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:30.191074 21037 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:30.307387 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 119990us
I20260317 16:18:30.308149 20162 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:30.245806 20041 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:30.357010 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 48950us
I20260317 16:18:30.357923 20162 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:30.359730 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.360462 20162 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:30.309983 20016 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:30.362210 20050 raft_consensus.cc:2410] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:30.311991 20198 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:30.176039 21196 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.313071 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.238034 21165 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.313704 21241 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:30.315752 21201 consensus_queue.cc:237] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
@ 0x2cd620 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
I20260317 16:18:30.264330 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
@ 0x2cfb03 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20260317 16:18:30.266283 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.259526 21235 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.305603 21006 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.245350 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:30.245771 21253 log.cc:927] Time spent T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20260317 16:18:30.263945 21164 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:30.416544 21182 raft_consensus.cc:2804] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
W20260317 16:18:30.365942 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.368202 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.414279 21236 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.432260 20826 raft_consensus.cc:2749] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:30.425554 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 180747us
I20260317 16:18:30.431517 21208 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:30.306128 20973 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:30.312602 21231 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.345862 21216 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.434520 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.430999 21197 raft_consensus.cc:2749] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:30.346066 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:30.347167 20892 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:30.356326 21156 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.414716 21154 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:30.507529 21154 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:30.507325 20271 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:30.508175 21154 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:30.509255 20321 tablet_bootstrap.cc:654] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:30.415056 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 105131us
W20260317 16:18:30.372967 20090 leader_election.cc:343] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:30.511305 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 138419us
W20260317 16:18:30.512768 20090 leader_election.cc:336] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Timed out: RequestConsensusVote RPC to 127.16.123.195:36885 timed out after 1.858s (SENT)
I20260317 16:18:30.513275 20090 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:30.512552 20016 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:30.443954 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.449457 20924 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.451140 21235 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.451623 21215 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:30.452587 21037 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.453444 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.453713 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:30.417618 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 57411.6us
I20260317 16:18:30.465353 21182 raft_consensus.cc:697] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:30.425597 20322 ts_tablet_manager.cc:1434] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.437s user 0.011s sys 0.002s
W20260317 16:18:30.532497 20322 ts_tablet_manager.cc:1467] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: Tablet startup took 544ms
I20260317 16:18:30.357960 21139 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:30.349422 21036 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:30.358831 21135 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:30.537839 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 66130us
W20260317 16:18:30.474983 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 470227us
I20260317 16:18:30.496003 20198 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:30.454125 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:30.468343 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.443754 20044 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:30.520192 20090 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:30.520869 21127 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:30.457038 20236 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:30.546469 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 89569.3us
I20260317 16:18:30.519451 20935 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.535460 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 191) took 1162 ms. Trace:
I20260317 16:18:30.507371 21156 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
W20260317 16:18:30.531522 20049 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 146) took 3296 ms (client timeout 1677 ms). Trace:
I20260317 16:18:30.466095 21236 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:30.534336 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.519387 21207 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.522396 21216 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:30.555372 20272 rpcz_store.cc:276] 0317 16:18:29.372636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.373713 (+ 1077us) service_pool.cc:225] Handling call
0317 16:18:29.374113 (+ 400us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:29.782926 (+408813us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:30.529129 (+746203us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400418a54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:30.535375 (+ 6246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938554752}
W20260317 16:18:30.556620 20049 rpcz_store.cc:269] 0317 16:18:27.235170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:27.258900 (+ 23730us) service_pool.cc:225] Handling call
0317 16:18:28.285201 (+1026301us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:30.531320 (+2246119us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b540003a7d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:30.531443 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1689530624}
I20260317 16:18:30.566681 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.557230 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:30.532809 20322 ts_tablet_manager.cc:1469] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321: Trace:
0317 16:18:05.183728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:05.198626 (+ 14898us) service_pool.cc:225] Handling call
0317 16:18:05.201449 (+ 2823us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:05.201547 (+ 98us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.360464 (+158917us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.404562 (+ 44098us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.405390 (+ 828us) inbound_call.cc:173] Queueing success response
0317 16:18:28.730400 (+23325010us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:29.988431 (+1258031us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:30.084001 (+ 95570us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:30.084567 (+ 566us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":19320,"raft.queue_time_us":68031,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":30990,"tablet-open.queue_time_us":23296906}
I20260317 16:18:30.478618 21231 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.570652 20199 raft_consensus.cc:2393] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:30.537082 21220 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.571913 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.572782 20320 tablet_bootstrap.cc:492] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:30.547922 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.555388 20892 consensus_queue.cc:1048] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
I20260317 16:18:30.604827 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:30.608325 20322 tablet_bootstrap.cc:492] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321: Bootstrap starting.
I20260317 16:18:30.605855 20049 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
W20260317 16:18:30.584141 20236 leader_election.cc:343] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:30.589359 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.568403 21216 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.610852 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:30.568249 20118 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 154) took 2398 ms. Trace:
I20260317 16:18:30.566339 20826 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.525403 21211 consensus_queue.cc:1048] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260317 16:18:30.614521 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.572988 21138 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:30.563356 20935 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.573374 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.606029 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.614010 21216 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:30.611621 21068 raft_consensus.cc:2749] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:30.635371 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.631198 21169 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:30.639468 21260 log.cc:927] Time spent T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260317 16:18:30.643996 21236 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.645589 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.653096 20162 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:30.611893 21201 raft_consensus.cc:2804] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:30.688112 21201 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.668879 20090 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:30.689083 21080 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:30.626360 21165 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.630722 20119 raft_consensus.cc:1275] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:30.689589 21006 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.690182 21267 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:30.690325 21006 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:30.688548 20162 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:30.691845 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.567973 21207 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.695750 20198 raft_consensus.cc:2393] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:30.690871 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.635265 21196 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.647826 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:30.702976 20050 raft_consensus.cc:2393] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:30.669376 21209 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.677037 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.685917 21135 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.688573 21201 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:30.622189 20826 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.713603 20318 tablet_bootstrap.cc:492] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:30.715188 20318 ts_tablet_manager.cc:1403] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.752s user 0.036s sys 0.036s
I20260317 16:18:30.630968 21138 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.690668 21267 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:30.714622 20162 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:30.706133 21005 consensus_meta.cc:287] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.697s user 0.010s sys 0.004s
I20260317 16:18:30.612690 20118 rpcz_store.cc:276] 0317 16:18:28.154731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.271168 (+116437us) service_pool.cc:225] Handling call
0317 16:18:28.271607 (+ 439us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:29.020419 (+748812us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:30.534220 (+1513801us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002e1654. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:30.552756 (+ 18536us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532418432}
I20260317 16:18:30.693385 21127 consensus_queue.cc:1059] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260317 16:18:30.722934 20239 leader_election.cc:343] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:30.698983 21068 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:30.711053 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.728070 20239 leader_election.cc:304] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:30.707820 20892 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:30.714425 21220 raft_consensus.cc:493] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.732548 20272 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:30.621712 21182 consensus_queue.cc:237] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.713023 20042 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:30.723295 21267 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:30.751333 21261 raft_consensus.cc:2749] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:30.757122 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.759132 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.760408 20124 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:30.763036 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.767897 20236 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:30.768697 20016 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:30.718218 21135 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:30.771807 21266 log.cc:927] Time spent T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260317 16:18:30.711880 21209 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.772624 21138 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.762370 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.732784 21036 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.786960 21269 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.791574 20124 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:30.799410 20016 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:30.806888 21236 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:30.807303 21236 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.807984 21236 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:30.787670 21165 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:30.811844 21165 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.811925 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.812378 21165 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:30.737352 21068 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.763991 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.738641 21220 raft_consensus.cc:515] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.825518 20320 tablet_bootstrap.cc:654] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:30.788570 21261 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.832024 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.741959 20892 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.835805 20892 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.842065 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.844122 20318 raft_consensus.cc:359] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.845079 20318 raft_consensus.cc:385] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:30.845358 20318 raft_consensus.cc:740] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:30.821746 21145 raft_consensus.cc:2804] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:30.855064 21145 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.855473 21145 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:30.856452 21220 leader_election.cc:290] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.829522 21269 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.820947 21231 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:30.852742 21068 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.853425 21006 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.854419 21037 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.850521 20195 raft_consensus.cc:1240] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:30.826007 21222 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:30.870383 21225 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.870644 21275 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.864190 20935 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.866675 20318 consensus_queue.cc:260] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:30.847330 21261 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.818202 20938 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260317 16:18:30.903429 21270 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:30.904052 21270 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.904609 20318 ts_tablet_manager.cc:1434] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.189s user 0.008s sys 0.001s
I20260317 16:18:30.905745 20318 tablet_bootstrap.cc:492] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:30.906477 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 239) took 1024 ms. Trace:
I20260317 16:18:30.899300 21275 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:30.908192 21272 log.cc:927] Time spent T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20260317 16:18:30.888964 21269 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.873812 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.914340 20124 raft_consensus.cc:2393] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 4: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:30.865499 21215 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.854312 20271 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:30.866673 20041 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:30.933449 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:30.935365 21273 log.cc:927] Time spent T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
W20260317 16:18:30.901433 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.886296 21037 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:30.837054 20190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 188) took 2336 ms (client timeout 1858 ms). Trace:
I20260317 16:18:30.847865 21277 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:30.917259 21275 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:30.894152 21068 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:30.897233 21072 consensus_meta.cc:287] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.565s user 0.012s sys 0.002s
I20260317 16:18:30.894205 21169 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.940346 20236 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:30.944337 20322 tablet_bootstrap.cc:654] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:30.954633 20091 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:30.896602 21268 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:30.907255 20193 rpcz_store.cc:276] 0317 16:18:29.882183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.991554 (+109371us) service_pool.cc:225] Handling call
0317 16:18:29.991877 (+ 323us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.086444 (+ 94567us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:30.086480 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:30.086523 (+ 43us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:30.086561 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:30.905941 (+819380us) spinlock_profiling.cc:243] Waited 94.4 ms on lock 0x7b540005b9d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:30.906028 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:30.906266 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217058048}
I20260317 16:18:30.912115 21270 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:30.949033 21277 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:30.961109 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 228) took 1337 ms. Trace:
W20260317 16:18:30.943193 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:30.952869 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:30.938840 20826 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:30.925158 21261 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:30.966638 20236 leader_election.cc:343] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:30.965196 20049 rpcz_store.cc:276] 0317 16:18:29.623301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.612828 (+989527us) service_pool.cc:225] Handling call
0317 16:18:30.631752 (+ 18924us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:30.632126 (+ 374us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:30.632154 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:30.670376 (+ 38222us) log.cc:844] Serialized 26 byte log entry
0317 16:18:30.799480 (+129104us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:30.959053 (+159573us) log.cc:844] Serialized 14 byte log entry
0317 16:18:30.960666 (+ 1613us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:30.960773 (+ 107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:30.960846 (+ 73us) raft_consensus.cc:1685] finished
0317 16:18:30.960866 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:30.961037 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1328,"raft.run_cpu_time_us":609,"raft.run_wall_time_us":30404,"thread_start_us":263971,"threads_started":2,"wal-append.queue_time_us":174740}
I20260317 16:18:30.895435 21222 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.028188 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.032805 20015 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.033465 20124 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:30.962512 21127 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:31.035151 20239 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:31.060252 21222 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.060325 21127 raft_consensus.cc:697] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:30.947885 21215 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.060997 21222 raft_consensus.cc:2804] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:31.061398 21222 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:31.061789 21222 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:31.061523 21127 consensus_queue.cc:237] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.062414 21215 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.032346 21135 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.046845 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:30.979611 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:31.067287 21278 log.cc:927] Time spent T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260317 16:18:31.070482 20238 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.073294 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:31.035483 21154 consensus_meta.cc:287] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.527s user 0.014s sys 0.000s
I20260317 16:18:30.989289 21207 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.050871 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:31.051354 20973 consensus_meta.cc:287] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.008s
I20260317 16:18:31.032341 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.054147 20826 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:30.982827 21275 raft_consensus.cc:2804] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:30.961110 21068 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:30.983196 21270 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:30.984311 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:30.963285 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:31.091418 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 396864us
I20260317 16:18:30.980208 21037 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.055953 21156 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.056269 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.051714 21110 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.057386 20015 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:31.093806 21209 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.058631 21211 raft_consensus.cc:3055] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:31.036522 21277 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:30.947396 20190 rpcz_store.cc:269] 0317 16:18:28.491674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.493060 (+ 1386us) service_pool.cc:225] Handling call
0317 16:18:30.816209 (+2323149us) spinlock_profiling.cc:243] Waited 85.2 ms on lock 0x7b540005c3d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:30.827163 (+ 10954us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195880192}
W20260317 16:18:30.982142 21164 consensus_meta.cc:287] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.528s user 0.010s sys 0.002s
I20260317 16:18:31.064438 20124 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:30.937125 21225 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.050038 21236 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:31.074915 21241 consensus_meta.cc:287] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.643s user 0.010s sys 0.004s
I20260317 16:18:31.075994 20892 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:31.036511 21169 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 16:18:31.092960 21139 consensus_meta.cc:287] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.543s user 0.012s sys 0.000s
I20260317 16:18:30.981629 21216 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:31.116379 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 59488.9us
I20260317 16:18:31.142997 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.144420 20015 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.146030 20015 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.058234 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.079982 21154 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:31.119920 20238 leader_election.cc:343] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:31.121299 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 406742us
I20260317 16:18:31.121877 21037 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:31.150611 20238 leader_election.cc:304] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:31.094944 20826 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.092635 20973 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.125681 20041 raft_consensus.cc:2410] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260317 16:18:31.115612 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.147136 20049 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:31.118472 21207 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.125758 20016 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:31.097383 21275 raft_consensus.cc:697] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:31.122390 21211 raft_consensus.cc:740] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:31.098239 21270 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:31.126413 21165 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.129029 21145 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.158171 20892 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:31.159287 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.136437 21241 raft_consensus.cc:2749] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:31.150730 21139 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:31.168391 20162 leader_election.cc:336] T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.995s (SENT)
I20260317 16:18:31.154932 21164 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:31.155633 20015 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:31.241456 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 85921.7us
I20260317 16:18:31.242682 20015 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.243794 20015 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.157186 21199 raft_consensus.cc:2749] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:31.186300 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.153404 21135 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 16:18:31.200670 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 74993.7us
I20260317 16:18:31.251660 20236 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:31.217836 21275 consensus_queue.cc:237] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.216118 20973 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.209614 21165 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.219285 21068 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.219691 21037 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:31.219941 21270 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.166671 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 198) took 1369 ms. Trace:
I20260317 16:18:31.220913 21267 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.221376 20267 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:31.222088 21156 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.271466 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:31.271713 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 170007us
I20260317 16:18:31.222751 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.175818 21207 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.176119 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:31.223842 20270 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 128) took 2422 ms (client timeout 1930 ms). Trace:
I20260317 16:18:31.169922 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.224408 21169 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.162235 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.201799 21209 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.225667 21145 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.335165 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.336263 21210 consensus_queue.cc:1059] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:31.336681 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.337368 21210 raft_consensus.cc:3055] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:31.337742 21210 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:31.337992 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:31.279193 20270 rpcz_store.cc:269] 0317 16:18:28.801325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.809773 (+ 8448us) service_pool.cc:225] Handling call
0317 16:18:31.223628 (+2413855us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b5400775bd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.223769 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2008023424}
I20260317 16:18:31.339818 21267 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.339990 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.340174 21207 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.226166 20826 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.340850 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 200) took 1516 ms. Trace:
I20260317 16:18:31.268814 21211 consensus_queue.cc:260] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.227829 20195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 227) took 2085 ms. Trace:
I20260317 16:18:31.341225 20270 rpcz_store.cc:276] 0317 16:18:29.824860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.339716 (+1514856us) service_pool.cc:225] Handling call
0317 16:18:31.340790 (+ 1074us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.341897 20195 rpcz_store.cc:276] 0317 16:18:29.142212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.150915 (+ 8703us) service_pool.cc:225] Handling call
0317 16:18:30.826289 (+1675374us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.885619 (+ 59330us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.227546 (+341927us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540005c3d0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.227739 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3838053248}
I20260317 16:18:31.229656 21215 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.217036 20318 tablet_bootstrap.cc:654] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:31.230342 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.412032 20198 raft_consensus.cc:3060] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:31.231444 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 231) took 1607 ms. Trace:
I20260317 16:18:31.205499 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:31.212214 21279 log.cc:927] Time spent T 8720c3d1b3994abeba1e121c1cf99275 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260317 16:18:31.341677 21211 raft_consensus.cc:3060] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:31.183029 21280 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:31.414978 21280 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.216549 21216 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.210738 21139 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.238359 20193 raft_consensus.cc:2410] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260317 16:18:31.247951 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.207757 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:31.428062 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.428649 20116 raft_consensus.cc:3060] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:31.429529 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.430370 20195 raft_consensus.cc:2393] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:31.413810 20191 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:31.431891 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.432487 20191 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:31.433386 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.246722 21236 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.255059 20016 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:31.249326 21131 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:31.435277 21131 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:31.435676 21131 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:31.435557 21210 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:31.291914 21169 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.436882 21210 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:31.249606 21231 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:31.222651 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 223) took 1601 ms. Trace:
I20260317 16:18:31.439724 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.439424 20045 rpcz_store.cc:276] 0317 16:18:29.621538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.624825 (+ 3287us) service_pool.cc:225] Handling call
0317 16:18:29.866852 (+242027us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:29.945029 (+ 78177us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.051146 (+106117us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.222417 (+1171271us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b5400729550. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.222584 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191200768}
I20260317 16:18:31.441378 20041 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:31.253146 21224 raft_consensus.cc:2749] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:31.236137 20892 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.260727 21219 raft_consensus.cc:2804] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:31.261238 21037 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:31.276691 21182 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.035s sys 0.076s
I20260317 16:18:31.283896 20321 tablet_bootstrap.cc:492] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:31.443770 21037 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:31.444298 20321 ts_tablet_manager.cc:1403] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.348s user 0.041s sys 0.025s
I20260317 16:18:31.264873 21135 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:31.447100 21135 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
@ 0x7fea3970a980 (unknown) at ??:0
I20260317 16:18:31.449622 20892 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.291014 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.450713 20049 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:31.450940 21277 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.451733 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.452461 20049 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:31.453238 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.334010 20322 tablet_bootstrap.cc:492] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321: No bootstrap required, opened a new log
I20260317 16:18:31.162613 20120 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:31.454161 20322 ts_tablet_manager.cc:1403] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321: Time spent bootstrapping tablet: real 0.866s user 0.029s sys 0.021s
W20260317 16:18:31.226307 21244 log.cc:927] Time spent T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 1.377s user 0.008s sys 0.011s
I20260317 16:18:31.342779 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.456529 20190 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:31.434432 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.457238 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 189) took 2085 ms. Trace:
I20260317 16:18:31.262293 20264 rpcz_store.cc:276] 0317 16:18:29.797032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.094196 (+1297164us) service_pool.cc:225] Handling call
0317 16:18:31.166530 (+ 72334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.459460 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.460567 20264 raft_consensus.cc:2393] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:31.343726 20117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 175) took 2399 ms. Trace:
I20260317 16:18:31.343604 21209 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.342506 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.345603 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.467000 21169 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.457872 20266 rpcz_store.cc:276] 0317 16:18:29.371779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.372116 (+ 337us) service_pool.cc:225] Handling call
0317 16:18:29.372530 (+ 414us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:29.372931 (+ 401us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:29.372973 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:29.373652 (+ 679us) log.cc:844] Serialized 26 byte log entry
0317 16:18:29.637539 (+263887us) spinlock_profiling.cc:243] Waited 8.29 ms on lock 0x7b18000b6560. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36234d17 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea36251b92 00007fea38e995ab 00007fea38e9f0c3 00007fea38e9f471 00007fea392bb93f
0317 16:18:29.721616 (+ 84077us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:29.723248 (+ 1632us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:29.723350 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:31.225711 (+1502361us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:31.456904 (+231193us) raft_consensus.cc:1685] finished
0317 16:18:31.456951 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.457154 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7373,"raft.queue_time_us":249,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":21896,"spinlock_wait_cycles":25718016,"thread_start_us":70765,"threads_started":1,"wal-append.queue_time_us":387455}
I20260317 16:18:31.487574 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:31.360049 20091 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:31.489037 20120 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:31.489780 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 129990us
I20260317 16:18:31.491287 21127 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:31.340003 20320 tablet_bootstrap.cc:492] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:31.493046 20320 ts_tablet_manager.cc:1403] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.940s user 0.022s sys 0.022s
I20260317 16:18:31.493168 21277 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.483230 21216 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:31.488042 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:31.499212 21201 consensus_meta.cc:287] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.776s user 0.013s sys 0.002s
I20260317 16:18:31.413251 20050 rpcz_store.cc:276] 0317 16:18:29.624327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.367722 (+743395us) service_pool.cc:225] Handling call
0317 16:18:31.231226 (+863504us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b540020a654. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.231381 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":591885184}
W20260317 16:18:31.507161 20048 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 157) took 2694 ms (client timeout 1995 ms). Trace:
I20260317 16:18:31.507460 20320 raft_consensus.cc:359] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.507790 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.508801 20050 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:31.277779 21241 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:31.338694 20194 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:31.332610 21154 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.451407 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.521670 20195 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:31.458380 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.343313 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:31.466141 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.434223 21225 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.468745 20321 raft_consensus.cc:359] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:31.478307 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 223314us
I20260317 16:18:31.548273 20050 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:31.548503 20162 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:31.549458 20050 raft_consensus.cc:1240] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:31.484922 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.467976 20117 rpcz_store.cc:276] 0317 16:18:28.944045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.291998 (+347953us) service_pool.cc:225] Handling call
0317 16:18:29.342885 (+ 50887us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:30.063124 (+720239us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.343236 (+1280112us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54001e9e54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:31.343642 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1382844288}
I20260317 16:18:31.352329 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:31.468158 21219 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:31.546909 20269 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:31.420606 21270 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:31.556990 20892 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:31.557164 21219 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:31.557677 21270 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.558522 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 177) took 1933 ms. Trace:
I20260317 16:18:31.558986 20045 rpcz_store.cc:276] 0317 16:18:29.624925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.539006 (+1914081us) service_pool.cc:225] Handling call
0317 16:18:31.557940 (+ 18934us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.558134 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.558166 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.558222 (+ 56us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.558255 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.558320 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.558460 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.562078 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.449821 21135 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:31.412338 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 108229us
I20260317 16:18:31.563438 20271 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:31.519629 21241 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:31.564559 21270 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.565845 21241 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.526150 20322 raft_consensus.cc:359] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.530308 20015 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:31.568404 20322 raft_consensus.cc:385] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:31.570307 20322 raft_consensus.cc:740] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Initialized, Role: FOLLOWER
I20260317 16:18:31.549835 20236 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:31.571450 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.571358 20322 consensus_queue.cc:260] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.573010 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.573173 20198 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:31.573506 21135 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:31.574154 21135 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:31.553201 21154 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:31.574631 21135 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:31.576351 21154 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:31.576810 21154 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:31.577070 21139 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
W20260317 16:18:31.577653 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:31.574888 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.419884 21215 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.574213 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.583014 21068 consensus_queue.cc:1048] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260317 16:18:31.421522 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.613570 21131 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:31.552537 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 200310us
I20260317 16:18:31.499621 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.554791 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:31.616321 20045 raft_consensus.cc:1240] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
W20260317 16:18:31.617043 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 62397.8us
I20260317 16:18:31.621879 20269 raft_consensus.cc:2468] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:31.626962 21231 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:31.627067 20269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 197) took 1836 ms (client timeout 1764 ms). Trace:
I20260317 16:18:31.627714 21231 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:31.628070 20268 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 131) took 2275 ms (client timeout 2047 ms). Trace:
W20260317 16:18:31.628633 20268 rpcz_store.cc:269] 0317 16:18:29.352961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.353136 (+ 175us) service_pool.cc:225] Handling call
0317 16:18:31.627845 (+2274709us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b5400394254. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.628001 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572404608}
I20260317 16:18:31.630111 21231 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.508198 20320 raft_consensus.cc:385] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:31.630647 20320 raft_consensus.cc:740] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:31.630586 20016 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:31.631649 21196 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:31.631498 20320 consensus_queue.cc:260] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.633251 20320 ts_tablet_manager.cc:1434] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.140s user 0.007s sys 0.000s
I20260317 16:18:31.633682 20236 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:31.634192 20320 tablet_bootstrap.cc:492] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
W20260317 16:18:31.507679 20048 rpcz_store.cc:269] 0317 16:18:28.813193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.830843 (+ 17650us) service_pool.cc:225] Handling call
0317 16:18:29.327562 (+496719us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:31.506950 (+2179388us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001b49d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.507094 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269808256}
I20260317 16:18:31.634624 21068 raft_consensus.cc:2804] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:31.635174 21068 raft_consensus.cc:697] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:31.635507 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.635726 20199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 203) took 1206 ms. Trace:
I20260317 16:18:31.636062 21068 consensus_queue.cc:237] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.636294 20199 rpcz_store.cc:276] 0317 16:18:30.429562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.434662 (+ 5100us) service_pool.cc:225] Handling call
0317 16:18:31.635446 (+1200784us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b54004a8dd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.635650 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78694144}
I20260317 16:18:31.636840 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.637723 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.557327 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:31.492609 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 952117us
I20260317 16:18:31.556973 21169 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.569802 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:31.588084 20198 raft_consensus.cc:2468] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:31.550112 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 178) took 1924 ms. Trace:
I20260317 16:18:31.540095 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:31.533596 20267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 204) took 1372 ms (client timeout 1554 ms). Trace:
I20260317 16:18:31.598315 21139 raft_consensus.cc:2749] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:31.587945 21280 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.586067 21215 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.582736 20892 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:31.638324 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.602034 20322 ts_tablet_manager.cc:1434] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321: Time spent starting tablet: real 0.147s user 0.006s sys 0.003s
I20260317 16:18:31.599776 21037 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.603268 20120 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:31.529331 21201 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.589395 21209 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.550334 20321 raft_consensus.cc:385] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:31.623620 20190 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:31.623872 21216 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260317 16:18:31.625308 21222 consensus_meta.cc:287] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.563s user 0.010s sys 0.000s
I20260317 16:18:31.625409 21224 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.626034 21131 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.626708 20266 raft_consensus.cc:2393] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:31.584923 20049 raft_consensus.cc:2410] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260317 16:18:31.669147 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.648821 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 206) took 1456 ms. Trace:
W20260317 16:18:31.728746 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 232565us
I20260317 16:18:31.692008 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:31.724180 20162 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.633688 20125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 167) took 1430 ms. Trace:
I20260317 16:18:31.739122 20050 rpcz_store.cc:276] 0317 16:18:29.625223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.549002 (+1923779us) service_pool.cc:225] Handling call
0317 16:18:31.549298 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.549884 (+ 586us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.550063 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.737664 20321 raft_consensus.cc:740] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:31.627316 20238 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:31.761307 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 142545us
I20260317 16:18:31.760807 20125 rpcz_store.cc:276] 0317 16:18:30.202855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.239184 (+ 36329us) service_pool.cc:225] Handling call
0317 16:18:30.239420 (+ 236us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.465345 (+225925us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:30.465375 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:30.465424 (+ 49us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:30.465463 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.633296 (+1167833us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54000adf54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:31.633347 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.633500 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":494206208}
I20260317 16:18:31.759511 20321 consensus_queue.cc:260] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.638674 20199 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:31.762754 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.737341 20272 rpcz_store.cc:276] 0317 16:18:30.192324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.610195 (+417871us) service_pool.cc:225] Handling call
0317 16:18:31.648395 (+1038200us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400685454. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.648724 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":277293440}
W20260317 16:18:31.772202 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.773144 20050 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:31.773283 20165 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:31.765424 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.655380 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.651470 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.574402 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.555410 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.630141 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.700927 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.616340 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.804304 20199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 202) took 1375 ms. Trace:
I20260317 16:18:31.794231 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.863297 20199 rpcz_store.cc:276] 0317 16:18:30.429109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.777705 (+1348596us) service_pool.cc:225] Handling call
0317 16:18:31.803453 (+ 25748us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.803914 (+ 461us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.803950 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.803989 (+ 39us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:31.804019 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.804093 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.804241 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.805951 20320 tablet_bootstrap.cc:654] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:31.774159 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.763335 20238 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:31.865770 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.658741 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34240 (request call id 163) took 1894 ms. Trace:
W20260317 16:18:31.699611 20043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 183) took 2016 ms (client timeout 1542 ms). Trace:
I20260317 16:18:31.674083 20116 raft_consensus.cc:2468] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:31.672648 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 162) took 2133 ms. Trace:
I20260317 16:18:31.765056 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 170) took 1146 ms. Trace:
I20260317 16:18:31.737673 21216 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:31.866647 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.897272 20050 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:31.805111 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:31.833119 20088 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.753129 20998 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.801656 20091 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:31.685283 20123 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 209) took 1331 ms. Trace:
I20260317 16:18:31.901054 20272 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:31.877457 20125 raft_consensus.cc:2393] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:31.875731 20117 raft_consensus.cc:2368] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3.
I20260317 16:18:31.795387 20271 raft_consensus.cc:2393] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:31.657122 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:31.627962 20269 rpcz_store.cc:269] 0317 16:18:29.790690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.956175 (+165485us) service_pool.cc:225] Handling call
0317 16:18:31.343137 (+1386962us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:31.626993 (+283856us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8341760}
I20260317 16:18:31.640913 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:31.675666 21242 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.654336 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:31.770444 21199 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:31.866531 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 103300us
W20260317 16:18:31.918236 20238 leader_election.cc:343] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:31.864807 21176 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:31.693012 20122 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 178) took 1776 ms (client timeout 1927 ms). Trace:
I20260317 16:18:31.934813 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.911765 21241 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:31.935135 20122 rpcz_store.cc:269] 0317 16:18:29.916927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.998650 (+ 81723us) service_pool.cc:225] Handling call
0317 16:18:31.692688 (+1694038us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b54006bd0d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.692916 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1067082624}
I20260317 16:18:31.936206 21241 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:31.937100 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:31.770368 21006 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:31.892452 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:31.849530 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 76321.6us
I20260317 16:18:31.752548 21222 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:31.792647 21118 raft_consensus.cc:493] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.891760 20199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 248) took 1428 ms. Trace:
I20260317 16:18:31.791373 21225 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.626726 20194 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
W20260317 16:18:31.902475 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 100925us
I20260317 16:18:31.775203 21210 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:31.883117 21291 log.cc:927] Time spent T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260317 16:18:31.770666 20048 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:31.955605 21210 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:31.905207 20321 ts_tablet_manager.cc:1434] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.460s user 0.013s sys 0.011s
I20260317 16:18:31.908473 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:31.957229 20321 ts_tablet_manager.cc:1467] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: Tablet startup took 512ms
I20260317 16:18:31.957437 20125 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:31.909403 20236 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:31.957526 20321 ts_tablet_manager.cc:1469] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b: Trace:
0317 16:18:04.984382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.984937 (+ 555us) service_pool.cc:225] Handling call
0317 16:18:05.010217 (+ 25280us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:05.017677 (+ 7460us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.227545 (+209868us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.259328 (+ 31783us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.259864 (+ 536us) inbound_call.cc:173] Queueing success response
0317 16:18:30.095718 (+24835854us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:31.444802 (+1349084us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:31.445384 (+ 582us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:31.445953 (+ 569us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":22673,"raft.queue_time_us":95970,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":18602,"tablet-open.queue_time_us":24835971}
W20260317 16:18:31.959141 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 49895.2us
I20260317 16:18:31.886832 21270 raft_consensus.cc:2804] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:31.960909 21270 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
W20260317 16:18:31.961202 21290 log.cc:927] Time spent T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.502s user 0.004s sys 0.000s
I20260317 16:18:31.961436 21270 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:31.883466 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:31.886117 21262 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:31.900882 20088 connection.cc:232] RPC call timeout handler was delayed by -1.2043996710001466s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1542s for a response
I20260317 16:18:31.964232 20125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 214) took 1240 ms. Trace:
I20260317 16:18:31.970435 20125 rpcz_store.cc:276] 0317 16:18:30.724430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.958131 (+1233701us) service_pool.cc:225] Handling call
0317 16:18:31.958410 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.958713 (+ 303us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.958740 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.963923 (+ 5183us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:31.963962 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.964044 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.964179 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.756063 21224 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.913009 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.886662 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 230) took 2259 ms. Trace:
I20260317 16:18:31.886998 21164 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:31.764760 21169 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:31.738068 20267 rpcz_store.cc:269] 0317 16:18:30.157235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.231228 (+ 73993us) service_pool.cc:225] Handling call
0317 16:18:31.476934 (+1245706us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b5400395654. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.529660 (+ 52726us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1214759680}
I20260317 16:18:31.904209 21216 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:31.848630 21135 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:31.981653 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.982359 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:31.983269 21135 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.983716 20264 raft_consensus.cc:2368] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:31.919946 20050 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:31.920882 20238 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:31.984935 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.974681 20042 rpcz_store.cc:276] 0317 16:18:29.624065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.812924 (+1188859us) service_pool.cc:225] Handling call
0317 16:18:30.813369 (+ 445us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.813671 (+ 302us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:31.265901 (+452230us) log.cc:844] Serialized 14 byte log entry
0317 16:18:31.735501 (+469600us) log.cc:844] Serialized 14 byte log entry
0317 16:18:31.771034 (+ 35533us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.805858 (+ 34824us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.817292 (+ 11434us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.847832 (+ 30540us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.883533 (+ 35701us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72638,"prepare.queue_time_us":374965,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":118,"raft.queue_time_us":15567,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":26934,"spinlock_wait_cycles":19063680,"thread_start_us":686279,"threads_started":2,"wal-append.queue_time_us":331196}
I20260317 16:18:31.985824 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.921584 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.916703 20192 raft_consensus.cc:2410] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260317 16:18:31.904296 20123 rpcz_store.cc:276] 0317 16:18:30.353672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.356027 (+ 2355us) service_pool.cc:225] Handling call
0317 16:18:30.440532 (+ 84505us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:30.473911 (+ 33379us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:30.478324 (+ 4413us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:30.556551 (+ 78227us) log.cc:844] Serialized 26 byte log entry
0317 16:18:31.684150 (+1127599us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b180036c460. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea38ea785c 00007fea3623539b 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea36251b92 00007fea38e995ab 00007fea38e9f0c3 00007fea38e9f471
0317 16:18:31.684462 (+ 312us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.684523 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.684753 (+ 230us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:31.684832 (+ 79us) raft_consensus.cc:1685] finished
0317 16:18:31.684847 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.685208 (+ 361us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8854784,"thread_start_us":97510,"threads_started":1,"wal-append.queue_time_us":135507}
I20260317 16:18:31.887077 20119 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:31.889621 20121 rpcz_store.cc:276] 0317 16:18:29.539335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.896237 (+356902us) service_pool.cc:225] Handling call
0317 16:18:29.896598 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:30.024714 (+128116us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:30.046046 (+ 21332us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:30.052777 (+ 6731us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:30.052843 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.672196 (+1619353us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54006bc6d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:31.672284 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.672549 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":274878720}
I20260317 16:18:31.987768 20119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 186) took 1434 ms. Trace:
I20260317 16:18:31.894381 20191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55456 (request call id 249) took 1351 ms. Trace:
I20260317 16:18:31.910851 21154 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:31.988391 20044 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 161) took 2246 ms (client timeout 2095 ms). Trace:
I20260317 16:18:31.988649 20191 rpcz_store.cc:276] 0317 16:18:30.542631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.574113 (+1031482us) service_pool.cc:225] Handling call
0317 16:18:31.893379 (+319266us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:31.988799 20044 rpcz_store.cc:269] 0317 16:18:29.741567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.092099 (+350532us) service_pool.cc:225] Handling call
0317 16:18:31.988218 (+1896119us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b5400244ad4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.988329 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":791563648}
I20260317 16:18:31.990214 21154 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:31.908910 20998 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:31.967332 21225 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.992431 20998 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.925272 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 207) took 1428 ms. Trace:
I20260317 16:18:31.993440 21225 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.988181 20119 rpcz_store.cc:276] 0317 16:18:30.554212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.650684 (+1096472us) service_pool.cc:225] Handling call
0317 16:18:31.987702 (+337018us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.993923 20265 raft_consensus.cc:2393] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:31.925716 21201 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.927822 20197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 211) took 1350 ms. Trace:
I20260317 16:18:31.993460 20193 rpcz_store.cc:276] 0317 16:18:30.496838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.923939 (+1427101us) service_pool.cc:225] Handling call
0317 16:18:31.924358 (+ 419us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.924842 (+ 484us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.924890 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.924940 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.924986 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.925057 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.925212 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.994894 20197 rpcz_store.cc:276] 0317 16:18:30.576994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.926643 (+1349649us) service_pool.cc:225] Handling call
0317 16:18:31.927103 (+ 460us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.927352 (+ 249us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.927393 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.927472 (+ 79us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.927521 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.927612 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.927750 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.996917 20197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 259) took 1065 ms. Trace:
I20260317 16:18:31.928349 20190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 247) took 1465 ms. Trace:
I20260317 16:18:31.997849 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:31.997997 20190 rpcz_store.cc:276] 0317 16:18:30.462920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.806730 (+1343810us) service_pool.cc:225] Handling call
0317 16:18:31.884326 (+ 77596us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.893590 (+ 9264us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.893649 (+ 59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.903611 (+ 9962us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.906900 (+ 3289us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.915000 (+ 8100us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.928283 (+ 13283us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:31.999953 20265 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 165) took 2068 ms (client timeout 1927 ms). Trace:
I20260317 16:18:32.000365 20190 raft_consensus.cc:1240] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:32.001662 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:31.932535 21219 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.933743 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.907608 21211 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:31.941510 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:31.943818 20199 rpcz_store.cc:276] 0317 16:18:30.463351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.864833 (+1401482us) service_pool.cc:225] Handling call
0317 16:18:31.890330 (+ 25497us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.890700 (+ 370us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.890726 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.890791 (+ 65us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:31.890822 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.890890 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.891699 (+ 809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.951234 21241 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.951761 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 181) took 2324 ms. Trace:
I20260317 16:18:31.876015 20273 rpcz_store.cc:276] 0317 16:18:29.763906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.781643 (+ 17737us) service_pool.cc:225] Handling call
0317 16:18:29.860121 (+ 78478us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:29.989708 (+129587us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.657673 (+1667965us) spinlock_profiling.cc:243] Waited 76.3 ms on lock 0x7b5400020150. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.658653 (+ 980us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175460096}
I20260317 16:18:31.920135 21199 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:31.939170 21006 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:31.953001 21222 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.948727 21118 raft_consensus.cc:515] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:31.954341 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 213) took 1237 ms. Trace:
I20260317 16:18:32.033161 20124 rpcz_store.cc:276] 0317 16:18:30.716485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.951032 (+1234547us) service_pool.cc:225] Handling call
0317 16:18:31.953521 (+ 2489us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.953806 (+ 285us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.953833 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.953882 (+ 49us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.953916 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.953977 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.954206 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.956543 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:31.892421 20266 rpcz_store.cc:276] 0317 16:18:30.633790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.499096 (+865306us) service_pool.cc:225] Handling call
0317 16:18:31.764045 (+264949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.005298 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:31.973210 20272 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:31.972672 21037 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:31.976110 21164 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:31.977041 20047 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 185) took 2255 ms (client timeout 2093 ms). Trace:
I20260317 16:18:32.037536 20120 raft_consensus.cc:2393] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:31.977051 21224 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:31.977728 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.977195 21169 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:31.986060 20050 raft_consensus.cc:2393] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:32.043887 21139 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:31.883220 20043 rpcz_store.cc:269] 0317 16:18:29.683243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.714955 (+ 31712us) service_pool.cc:225] Handling call
0317 16:18:31.699322 (+1984367us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b54001dae54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.699516 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119120128}
I20260317 16:18:31.988174 20194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 250) took 1296 ms. Trace:
I20260317 16:18:31.970551 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.004267 20239 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:31.969856 20196 raft_consensus.cc:2393] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
W20260317 16:18:32.007230 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 65828.4us
I20260317 16:18:32.081887 20165 leader_election.cc:400] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:32.008275 20049 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:31.984799 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 64042.8us
I20260317 16:18:31.986406 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.026962 20321 tablet_bootstrap.cc:492] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:32.084502 20238 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:31.973310 20892 raft_consensus.cc:2804] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:32.029942 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:32.087487 20199 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:31.988850 20192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34608 (request call id 171) took 1025 ms. Trace:
W20260317 16:18:32.012033 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:31.971673 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.044459 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 167) took 1607 ms. Trace:
I20260317 16:18:32.092545 20272 rpcz_store.cc:276] 0317 16:18:30.451564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.902362 (+1450798us) service_pool.cc:225] Handling call
0317 16:18:32.044370 (+142008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.997274 20197 rpcz_store.cc:276] 0317 16:18:30.931883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.996060 (+1064177us) service_pool.cc:225] Handling call
0317 16:18:31.996356 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.996570 (+ 214us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.996597 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.996639 (+ 42us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:31.996675 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.996738 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.996859 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:31.990242 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.005235 20190 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:32.023362 21280 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:32.031291 20041 consensus_meta.cc:287] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.520s user 0.007s sys 0.005s
I20260317 16:18:32.051322 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.025384 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:31.977144 21235 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:32.024399 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:32.116232 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:32.029704 21199 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:32.123601 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.124598 20122 raft_consensus.cc:2410] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:32.125941 20122 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
@ 0x7fea32b04e87 gsignal at ??:0
I20260317 16:18:32.101955 20042 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:32.130714 20117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 192) took 1066 ms. Trace:
I20260317 16:18:32.131410 20117 rpcz_store.cc:276] 0317 16:18:31.064083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.556996 (+492913us) service_pool.cc:225] Handling call
0317 16:18:32.130293 (+573297us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b5400111554. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:32.130596 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64699264}
I20260317 16:18:32.132303 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.133374 20042 raft_consensus.cc:2393] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:32.112133 21068 raft_consensus.cc:2804] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:32.135596 20119 raft_consensus.cc:1275] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:32.136446 20119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 220) took 1190 ms. Trace:
I20260317 16:18:32.135300 21068 raft_consensus.cc:697] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:32.136790 20119 rpcz_store.cc:276] 0317 16:18:30.946357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.994328 (+1047971us) service_pool.cc:225] Handling call
0317 16:18:32.134724 (+140396us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:32.135056 (+ 332us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.136136 (+ 1080us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.136386 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25082624}
W20260317 16:18:32.139321 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:32.045678 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.049057 20195 raft_consensus.cc:2393] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:31.987156 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:32.059456 20047 rpcz_store.cc:269] 0317 16:18:29.721210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.088327 (+367117us) service_pool.cc:225] Handling call
0317 16:18:31.976845 (+1888518us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54001b58d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.976969 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":567713536}
I20260317 16:18:32.003948 21262 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:32.165162 21216 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:32.165660 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.070708 20267 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
W20260317 16:18:32.078218 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 74032.3us
I20260317 16:18:32.167294 21242 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:32.110303 21280 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:32.168124 21280 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:32.168773 20264 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:32.142393 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.082387 20165 leader_election.cc:403] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260317 16:18:32.117102 20044 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
W20260317 16:18:32.171320 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 89795.3us
I20260317 16:18:32.171425 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.172120 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.172402 20043 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:32.172631 20165 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:32.172631 21262 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:32.089071 20015 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:32.090581 20238 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:32.000407 20265 rpcz_store.cc:269] 0317 16:18:29.945879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:29.983136 (+ 37257us) service_pool.cc:225] Handling call
0317 16:18:31.654099 (+1670963us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:31.999727 (+345628us) spinlock_profiling.cc:243] Waited 78.4 ms on lock 0x7b54003ba1d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:31.999882 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185653888}
I20260317 16:18:32.097106 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:32.074066 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:32.107946 20125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 221) took 1044 ms. Trace:
I20260317 16:18:32.026103 20045 rpcz_store.cc:276] 0317 16:18:29.626074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.560016 (+1933942us) service_pool.cc:225] Handling call
0317 16:18:31.560350 (+ 334us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.617127 (+ 56777us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.917378 (+300251us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b540061b054. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:31.950203 (+ 32825us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127995776}
W20260317 16:18:32.195657 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 105210us
I20260317 16:18:32.196293 20049 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:32.196652 20238 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:32.083037 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:32.091245 20192 rpcz_store.cc:276] 0317 16:18:30.989149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.654981 (+665832us) service_pool.cc:225] Handling call
0317 16:18:31.988781 (+333800us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.120980 21006 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:32.124421 20046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 233) took 2230 ms (client timeout 2966 ms). Trace:
W20260317 16:18:32.242910 20046 rpcz_store.cc:269] 0317 16:18:29.893506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.919403 (+2025897us) service_pool.cc:225] Handling call
0317 16:18:32.124356 (+204953us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.244292 20046 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260317 16:18:32.043040 20270 raft_consensus.cc:2393] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:32.141513 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:31.997898 21210 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:32.045325 21072 raft_consensus.cc:2749] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:32.155225 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 222) took 1027 ms. Trace:
I20260317 16:18:32.246933 20124 rpcz_store.cc:276] 0317 16:18:31.128043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.118066 (+990023us) service_pool.cc:225] Handling call
0317 16:18:32.154461 (+ 36395us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.154683 (+ 222us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.154799 (+ 116us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.154856 (+ 57us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.154902 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.154980 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.155152 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.087173 20892 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:32.249402 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 52869.9us
W20260317 16:18:32.156438 20162 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.159970 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.114163 20196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34608 (request call id 166) took 1613 ms. Trace:
I20260317 16:18:32.163671 21164 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:32.166435 20047 raft_consensus.cc:2368] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 4. Current term is 5.
W20260317 16:18:32.173859 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 84899.4us
I20260317 16:18:32.172802 20044 raft_consensus.cc:2468] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:32.059973 20048 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:32.078409 21139 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:32.188516 20125 rpcz_store.cc:276] 0317 16:18:31.063268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.104123 (+1040855us) service_pool.cc:225] Handling call
0317 16:18:32.104844 (+ 721us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.105145 (+ 301us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.105190 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.105257 (+ 67us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:32.105304 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.105388 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.107388 (+ 2000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.197594 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.134457 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:32.199631 21211 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:32.200332 20197 raft_consensus.cc:1240] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260317 16:18:32.200958 21219 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:32.202253 20117 raft_consensus.cc:1275] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:32.204770 21115 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:32.211431 20198 raft_consensus.cc:1240] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:32.210462 21242 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:32.171562 20893 consensus_queue.cc:1048] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260317 16:18:32.112498 20041 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:32.213918 20998 raft_consensus.cc:2749] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:32.214805 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 216) took 1428 ms. Trace:
I20260317 16:18:32.216209 21118 leader_election.cc:290] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:32.141008 21235 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.153345 20116 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:32.077467 20194 rpcz_store.cc:276] 0317 16:18:30.692241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.954710 (+1262469us) service_pool.cc:225] Handling call
0317 16:18:31.987563 (+ 32853us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.987771 (+ 208us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:31.987809 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:31.987861 (+ 52us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:31.987913 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:31.987984 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:31.988118 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.173249 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:32.138273 21068 consensus_queue.cc:237] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:32.228650 21169 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:32.232841 20265 raft_consensus.cc:1240] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:32.195043 21241 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3.
I20260317 16:18:32.248207 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 173) took 1487 ms. Trace:
I20260317 16:18:32.296780 21222 consensus_queue.cc:1048] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260317 16:18:32.301474 20272 rpcz_store.cc:276] 0317 16:18:30.760812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.103415 (+1342603us) service_pool.cc:225] Handling call
0317 16:18:32.248149 (+144734us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.253296 20273 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:32.303382 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 174) took 1400 ms. Trace:
I20260317 16:18:32.303782 20273 rpcz_store.cc:276] 0317 16:18:30.902648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.059203 (+1156555us) service_pool.cc:225] Handling call
0317 16:18:32.303298 (+244095us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.287909 20271 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260317 16:18:32.250818 20892 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:32.259512 20015 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:32.309644 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 50424.4us
I20260317 16:18:32.258533 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 160) took 2515 ms. Trace:
I20260317 16:18:32.257606 21262 raft_consensus.cc:2804] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:32.260850 21006 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:32.264798 20123 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:32.279496 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 187) took 2376 ms. Trace:
I20260317 16:18:32.312075 20050 rpcz_store.cc:276] 0317 16:18:29.903313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.278313 (+2375000us) service_pool.cc:225] Handling call
0317 16:18:32.278746 (+ 433us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.279210 (+ 464us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.279428 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.313349 20050 raft_consensus.cc:1240] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:32.266993 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.296106 21169 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:32.305678 20048 raft_consensus.cc:1240] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:32.270782 20042 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:32.276037 21242 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:32.257138 20196 rpcz_store.cc:276] 0317 16:18:30.532048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:30.733359 (+201311us) service_pool.cc:225] Handling call
0317 16:18:31.656932 (+923573us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:32.110416 (+453484us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8024832}
I20260317 16:18:32.271085 21211 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.284446 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 247) took 1030 ms. Trace:
I20260317 16:18:32.305335 20273 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:32.254933 20238 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:32.324901 20043 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 4: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:32.287220 20197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 212) took 1007 ms. Trace:
I20260317 16:18:32.310727 20272 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260317 16:18:32.313946 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 170) took 2366 ms. Trace:
I20260317 16:18:32.314404 21262 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:32.386557 20050 rpcz_store.cc:276] 0317 16:18:29.947783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.312818 (+2365035us) service_pool.cc:225] Handling call
0317 16:18:32.313153 (+ 335us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.313710 (+ 557us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.313889 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:32.316902 20318 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.029s sys 0.042s
W20260317 16:18:32.287868 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 379323us
W20260317 16:18:32.388657 20318 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.892s user 0.029s sys 0.042s
I20260317 16:18:32.389868 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:32.351343 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 96501.3us
I20260317 16:18:32.352487 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 135) took 2252 ms. Trace:
I20260317 16:18:32.293766 21216 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:32.326174 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 203) took 2463 ms. Trace:
I20260317 16:18:32.313949 21164 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:32.297482 21199 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.331122 21169 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:32.392309 20273 rpcz_store.cc:276] 0317 16:18:30.100574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.304621 (+2204047us) service_pool.cc:225] Handling call
0317 16:18:32.305050 (+ 429us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.352111 (+ 47061us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.352419 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.388965 20893 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:32.393888 20016 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:32.374398 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.405470 20198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 246) took 2168 ms. Trace:
I20260317 16:18:32.334642 21224 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:32.335594 20122 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:32.350833 21225 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:32.323727 20044 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 162) took 2579 ms. Trace:
I20260317 16:18:32.389750 21262 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:32.429081 20198 rpcz_store.cc:276] 0317 16:18:30.236577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.759680 (+1523103us) service_pool.cc:225] Handling call
0317 16:18:32.211188 (+451508us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.277472 (+ 66284us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.405124 (+127652us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b5400459f50. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:32.405362 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":962952960}
I20260317 16:18:32.322266 21006 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:32.279783 20121 rpcz_store.cc:276] 0317 16:18:30.786696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.987896 (+1201200us) service_pool.cc:225] Handling call
0317 16:18:32.122001 (+134105us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:32.122326 (+ 325us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.122732 (+ 406us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.153181 (+ 30449us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.176021 (+ 22840us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.193259 (+ 17238us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.214528 (+ 21269us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.214708 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39929728}
I20260317 16:18:32.311723 20046 rpcz_store.cc:276] 0317 16:18:29.741191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.243705 (+2502514us) service_pool.cc:225] Handling call
0317 16:18:32.244046 (+ 341us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.246476 (+ 2430us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.256425 (+ 9949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.457983 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 136) took 2349 ms. Trace:
I20260317 16:18:32.474530 20273 rpcz_store.cc:276] 0317 16:18:30.108407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.421135 (+2312728us) service_pool.cc:225] Handling call
0317 16:18:32.457464 (+ 36329us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.457647 (+ 183us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.457682 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.457735 (+ 53us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:32.457765 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.457823 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.457934 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.399523 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 169) took 2655 ms. Trace:
I20260317 16:18:32.350595 20041 rpcz_store.cc:276] 0317 16:18:31.251889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.439305 (+187416us) service_pool.cc:225] Handling call
0317 16:18:32.282188 (+842883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.412418 20047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 232) took 2515 ms. Trace:
I20260317 16:18:32.425137 21275 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:32.483054 20318 tablet_bootstrap.cc:492] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:32.524390 20318 ts_tablet_manager.cc:1403] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 1.619s user 0.043s sys 0.042s
I20260317 16:18:32.426154 21277 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:32.410792 21135 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260317 16:18:32.488368 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.468870 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 253) took 1663 ms. Trace:
W20260317 16:18:32.528856 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 356277us
I20260317 16:18:32.386073 20197 rpcz_store.cc:276] 0317 16:18:31.279476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.113511 (+834035us) service_pool.cc:225] Handling call
0317 16:18:32.199978 (+ 86467us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.274142 (+ 74164us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.284815 (+ 10673us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.435995 21139 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:32.459080 21164 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:32.422935 21222 consensus_queue.cc:1059] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260317 16:18:32.421247 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:32.491489 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 97673.1us
I20260317 16:18:32.555647 20016 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:32.456007 21224 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:32.548810 20320 tablet_bootstrap.cc:492] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:32.549136 20165 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:32.476189 21145 raft_consensus.cc:493] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:32.552748 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 137) took 2416 ms. Trace:
I20260317 16:18:32.458025 21225 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.410605 20265 rpcz_store.cc:276] 0317 16:18:29.862860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.232143 (+2369283us) service_pool.cc:225] Handling call
0317 16:18:32.232565 (+ 422us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.304079 (+ 71514us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.323011 (+ 18932us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.594295 20320 ts_tablet_manager.cc:1403] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.960s user 0.027s sys 0.037s
I20260317 16:18:32.492738 20267 raft_consensus.cc:1240] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:32.566172 20016 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:32.593919 20273 rpcz_store.cc:276] 0317 16:18:30.136043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.495244 (+2359201us) service_pool.cc:225] Handling call
0317 16:18:32.552003 (+ 56759us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.552262 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.552304 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.552369 (+ 65us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.552434 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.552516 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.552671 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:32.602491 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 53384.6us
I20260317 16:18:32.523121 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:32.526618 21135 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:32.523298 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:32.627333 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.553808 21156 raft_consensus.cc:493] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:32.624138 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34240 (request call id 164) took 2859 ms. Trace:
I20260317 16:18:32.594142 21216 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:32.639360 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 134) took 2557 ms. Trace:
I20260317 16:18:32.647112 20272 rpcz_store.cc:276] 0317 16:18:30.081960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.302391 (+2220431us) service_pool.cc:225] Handling call
0317 16:18:32.307727 (+ 5336us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.391339 (+ 83612us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.638946 (+247607us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b54005b4d50. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:32.639268 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10896768}
I20260317 16:18:32.651615 20320 raft_consensus.cc:359] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.652864 20320 raft_consensus.cc:385] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:32.653824 20320 raft_consensus.cc:740] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:32.608549 21241 raft_consensus.cc:2764] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260317 16:18:32.664004 20320 consensus_queue.cc:260] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.550331 21277 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.436657 20044 rpcz_store.cc:276] 0317 16:18:29.741919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.266020 (+2524101us) service_pool.cc:225] Handling call
0317 16:18:32.278751 (+ 12731us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.287652 (+ 8901us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.292887 (+ 5235us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.294825 (+ 1938us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.299867 (+ 5042us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.308308 (+ 8441us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.320974 (+ 12666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.522691 20048 rpcz_store.cc:276] 0317 16:18:29.744834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.272516 (+2527682us) service_pool.cc:225] Handling call
0317 16:18:32.289994 (+ 17478us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.335374 (+ 45380us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.393416 (+ 58042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.628628 21037 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:32.661083 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.605537 20118 raft_consensus.cc:2468] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:32.700954 20090 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:32.628424 21135 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:32.551338 21275 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:32.734513 20090 leader_election.cc:304] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:32.617168 21222 raft_consensus.cc:3055] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:32.737960 20165 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:32.659232 21299 log.cc:927] Time spent T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.204s user 0.000s sys 0.001s
I20260317 16:18:32.619531 21145 raft_consensus.cc:515] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:32.749521 20198 raft_consensus.cc:3055] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:32.775494 20198 raft_consensus.cc:740] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:32.701238 21216 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:32.733577 21298 log.cc:927] Time spent T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.411s user 0.001s sys 0.001s
I20260317 16:18:32.780944 20165 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:32.789770 20320 ts_tablet_manager.cc:1434] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.189s user 0.013s sys 0.000s
I20260317 16:18:32.659263 21156 raft_consensus.cc:515] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:32.780074 20265 raft_consensus.cc:1275] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:32.790935 21139 consensus_queue.cc:1059] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:32.667189 21235 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:32.793107 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 227100us
I20260317 16:18:32.794844 20165 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:32.743166 21201 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:32.736631 21211 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:32.738945 21037 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:32.819598 21037 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:32.736141 21115 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:32.823277 21115 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:32.826042 21115 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:32.803978 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:32.777987 21176 consensus_meta.cc:287] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.835s user 0.021s sys 0.001s
I20260317 16:18:32.828526 21176 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:32.857564 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 166) took 3114 ms. Trace:
I20260317 16:18:32.858171 20049 rpcz_store.cc:276] 0317 16:18:29.743237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.778215 (+3034978us) service_pool.cc:225] Handling call
0317 16:18:32.856978 (+ 78763us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.857173 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.857200 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.857245 (+ 45us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.857274 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.857339 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.857495 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.763044 21222 raft_consensus.cc:740] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:32.526132 20047 rpcz_store.cc:276] 0317 16:18:29.893165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.276319 (+2383154us) service_pool.cc:225] Handling call
0317 16:18:32.307730 (+ 31411us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.322303 (+ 14573us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.331335 (+ 9032us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.335232 (+ 3897us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.350408 (+ 15176us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.390210 (+ 39802us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.408223 (+ 18013us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.780092 21210 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:32.794842 20198 consensus_queue.cc:260] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:32.786850 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:32.801052 21282 raft_consensus.cc:2804] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:32.873378 20198 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:32.801229 21199 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:32.891067 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 85302.9us
W20260317 16:18:32.901343 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 195907us
W20260317 16:18:32.906018 20015 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:32.885046 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 150589us
I20260317 16:18:32.910040 20049 raft_consensus.cc:1275] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:32.911109 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 245) took 1851 ms. Trace:
I20260317 16:18:32.911540 20049 rpcz_store.cc:276] 0317 16:18:31.060137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.872275 (+1812138us) service_pool.cc:225] Handling call
0317 16:18:32.909686 (+ 37411us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.910807 (+ 1121us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.911046 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.644666 20271 rpcz_store.cc:276] 0317 16:18:29.764246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.121580 (+2357334us) service_pool.cc:225] Handling call
0317 16:18:32.121984 (+ 404us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.310896 (+188912us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.623709 (+312813us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400541fd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:32.624049 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346556800}
I20260317 16:18:32.862576 21118 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:32.871134 21312 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:32.803333 21216 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:32.879530 21211 raft_consensus.cc:2804] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
W20260317 16:18:32.928865 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 124972us
I20260317 16:18:32.901859 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 207) took 2648 ms. Trace:
W20260317 16:18:32.900435 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 108607us
I20260317 16:18:32.932334 20165 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:32.933648 20016 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:32.944303 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:32.936328 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:32.936637 21275 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:32.902402 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56776 (request call id 214) took 1284 ms. Trace:
I20260317 16:18:32.908686 20192 raft_consensus.cc:2468] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:32.917325 21282 raft_consensus.cc:697] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
W20260317 16:18:32.940798 20190 consensus_meta.cc:287] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.824s user 0.015s sys 0.000s
W20260317 16:18:32.916436 20270 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 168) took 2461 ms (client timeout 2132 ms). Trace:
I20260317 16:18:32.941582 21327 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:32.902673 21006 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:32.952224 20198 raft_consensus.cc:1275] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260317 16:18:32.919930 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:32.920873 21303 log.cc:927] Time spent T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.398s user 0.000s sys 0.001s
I20260317 16:18:33.003504 20198 pending_rounds.cc:85] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting all ops after (but not including) 0
I20260317 16:18:33.003787 20198 pending_rounds.cc:107] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260317 16:18:33.004038 20198 raft_consensus.cc:2889] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260317 16:18:32.961011 21308 log.cc:927] Time spent T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260317 16:18:32.961206 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.962101 21225 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:32.931116 20225 tablet.cc:2379] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 16:18:32.965279 20016 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:32.968793 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 251) took 1351 ms. Trace:
I20260317 16:18:33.035503 20049 rpcz_store.cc:276] 0317 16:18:31.617805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.919380 (+1301575us) service_pool.cc:225] Handling call
0317 16:18:32.968068 (+ 48688us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.968352 (+ 284us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.968390 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.968445 (+ 55us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:32.968496 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.968570 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.968722 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:32.882073 21201 consensus_queue.cc:1059] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260317 16:18:32.915043 21176 consensus_queue.cc:1059] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260317 16:18:33.001561 20318 raft_consensus.cc:359] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.042197 20318 raft_consensus.cc:385] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:33.042570 20318 raft_consensus.cc:740] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:33.043481 20318 consensus_queue.cc:260] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:32.973186 21275 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:32.930631 21312 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:32.974854 21305 log.cc:927] Time spent T 43d7dcb7b21548889a850700415701a9 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260317 16:18:32.960251 21145 leader_election.cc:290] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:32.983215 20190 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:33.058444 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:33.059198 20049 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:33.028330 21211 raft_consensus.cc:697] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:32.970183 21115 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:33.008957 21313 log.cc:927] Time spent T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
W20260317 16:18:33.010646 21304 log.cc:927] Time spent T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20260317 16:18:33.011449 21156 leader_election.cc:290] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:33.011147 21110 consensus_queue.cc:1059] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:32.963268 20264 rpcz_store.cc:276] 0317 16:18:31.618203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.985480 (+367277us) service_pool.cc:225] Handling call
0317 16:18:32.879065 (+893585us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400718754. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:32.882706 (+ 3641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":362944384}
I20260317 16:18:32.548496 20193 rpcz_store.cc:276] 0317 16:18:30.793584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.995666 (+1202082us) service_pool.cc:225] Handling call
0317 16:18:31.996014 (+ 348us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:31.996248 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:31.996570 (+ 322us) log.cc:844] Serialized 14 byte log entry
0317 16:18:32.383183 (+386613us) log.cc:844] Serialized 14 byte log entry
0317 16:18:32.407279 (+ 24096us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.413488 (+ 6209us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:32.416951 (+ 3463us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.431674 (+ 14723us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:32.457268 (+ 25594us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12794,"prepare.queue_time_us":184092,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":23383,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":27460,"spinlock_wait_cycles":80468224,"thread_start_us":292570,"threads_started":1,"wal-append.queue_time_us":438667}
I20260317 16:18:33.073982 21154 raft_consensus.cc:3055] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:33.020608 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:32.943086 21222 consensus_queue.cc:260] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:32.943009 20265 rpcz_store.cc:276] 0317 16:18:30.233684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.617549 (+2383865us) service_pool.cc:225] Handling call
0317 16:18:32.698515 (+ 80966us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.801990 (+103475us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.882265 (+ 80275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.116772 21154 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:33.143481 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.144134 20265 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:33.143981 21154 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.082279 21337 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:32.977928 21210 raft_consensus.cc:2804] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:32.972508 21324 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:32.953670 21209 consensus_queue.cc:1059] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260317 16:18:33.026312 20321 tablet_bootstrap.cc:654] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:33.000972 21224 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:33.028342 20320 tablet_bootstrap.cc:492] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:32.875885 21037 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.032995 20090 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:33.034514 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:33.047724 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 82544.1us
I20260317 16:18:33.022583 21216 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:33.000044 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:33.005244 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:33.075420 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:33.161444 20116 consensus_meta.cc:287] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.867s user 0.012s sys 0.005s
I20260317 16:18:33.088429 20318 ts_tablet_manager.cc:1434] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.564s user 0.009s sys 0.001s
I20260317 16:18:33.091035 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:33.194043 20318 ts_tablet_manager.cc:1467] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: Tablet startup took 669ms
I20260317 16:18:33.093096 20239 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:33.073848 20893 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.003s sys 0.004s
W20260317 16:18:32.982446 20270 rpcz_store.cc:269] 0317 16:18:30.451967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.342212 (+890245us) service_pool.cc:225] Handling call
0317 16:18:32.880819 (+1538607us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b5400226854. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:32.908407 (+ 27588us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1215810304}
W20260317 16:18:33.019392 21315 log.cc:927] Time spent T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260317 16:18:33.065591 20190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34608 (request call id 183) took 1064 ms. Trace:
I20260317 16:18:33.090534 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.102449 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.106788 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:33.054960 21275 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:33.200768 20893 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.003s sys 0.005s
I20260317 16:18:33.107544 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.065243 21145 consensus_queue.cc:1048] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
I20260317 16:18:33.109090 21312 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.109656 21110 raft_consensus.cc:3055] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:33.200464 21156 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:33.111624 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 173) took 2802 ms. Trace:
I20260317 16:18:33.198207 20271 raft_consensus.cc:2376] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4. Re-sending same reply.
I20260317 16:18:33.115180 20191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 251) took 2408 ms. Trace:
I20260317 16:18:33.243373 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:33.244354 21242 consensus_meta.cc:287] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.855s user 0.012s sys 0.001s
I20260317 16:18:33.244568 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.244863 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.245955 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:33.245726 20271 raft_consensus.cc:2468] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:33.228175 20047 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:33.115262 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 237) took 2805 ms. Trace:
I20260317 16:18:33.087836 21115 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:33.117331 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260317 16:18:33.115844 21270 consensus_meta.cc:287] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 1.154s user 0.009s sys 0.004s
I20260317 16:18:33.172161 21211 consensus_queue.cc:237] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.251533 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.087922 21282 consensus_queue.cc:237] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.252801 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:33.254495 20270 raft_consensus.cc:1275] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:32.960335 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:33.180747 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 147835us
I20260317 16:18:33.181458 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.182487 20199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 176) took 1760 ms. Trace:
I20260317 16:18:33.146301 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:33.190866 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.055194 21327 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election won for term 4
W20260317 16:18:33.195879 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 102878us
W20260317 16:18:33.187044 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:33.194340 20318 ts_tablet_manager.cc:1469] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f: Trace:
0317 16:18:05.184511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:05.184682 (+ 171us) service_pool.cc:225] Handling call
0317 16:18:05.187177 (+ 2495us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:05.187272 (+ 95us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.280244 (+ 92972us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.314673 (+ 34429us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.315178 (+ 505us) inbound_call.cc:173] Queueing success response
0317 16:18:30.905300 (+25590122us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:32.524711 (+1619411us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:32.959160 (+434449us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:32.959801 (+ 641us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":11551,"raft.queue_time_us":10819,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":15573,"tablet-open.queue_time_us":25590216}
I20260317 16:18:33.255678 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 210) took 2788 ms. Trace:
I20260317 16:18:33.213621 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:33.186784 21329 raft_consensus.cc:2804] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:33.107542 20091 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:33.283784 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 176335us
I20260317 16:18:33.192461 21324 raft_consensus.cc:697] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:33.110577 21164 consensus_meta.cc:287] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.554s user 0.009s sys 0.000s
I20260317 16:18:33.200162 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 184) took 3494 ms. Trace:
I20260317 16:18:33.195354 20116 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
W20260317 16:18:33.291676 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 62836.8us
I20260317 16:18:33.180666 21337 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.114491 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 171) took 3110 ms. Trace:
I20260317 16:18:33.200999 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 133) took 3119 ms. Trace:
I20260317 16:18:33.227972 20893 raft_consensus.cc:697] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:33.232875 21110 raft_consensus.cc:740] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:33.235879 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.238355 21275 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:33.207803 20190 rpcz_store.cc:276] 0317 16:18:32.000885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.001400 (+ 515us) service_pool.cc:225] Handling call
0317 16:18:33.061109 (+1059709us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.241235 20320 tablet_bootstrap.cc:654] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:33.233341 21156 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.242218 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:33.242374 20238 leader_election.cc:343] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:33.317413 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 75078.1us
I20260317 16:18:33.242931 20191 rpcz_store.cc:276] 0317 16:18:30.707312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.989732 (+1282420us) service_pool.cc:225] Handling call
0317 16:18:31.990090 (+ 358us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:31.990518 (+ 428us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:31.997943 (+ 7425us) log.cc:844] Serialized 14 byte log entry
0317 16:18:32.157722 (+159779us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b18000bf9e0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36220386 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36252383 00007fea36254751
0317 16:18:32.158227 (+ 505us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b18000bf9e0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36252383 00007fea36254751 00007fea36254709 00007fea36254699 00007fea36254661 00007fea3625395d 0000000000379254 0000000000379099 00007fea3623574a
0317 16:18:32.407631 (+249404us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.523676 (+116045us) log.cc:844] Serialized 26 byte log entry
0317 16:18:32.553956 (+ 30280us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:32.593360 (+ 39404us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.605830 (+ 12470us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.114970 (+509140us) raft_consensus.cc:1685] finished
0317 16:18:33.114994 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.115121 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13749,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":2490,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":36060,"spinlock_wait_cycles":13049984,"thread_start_us":380863,"threads_started":1,"wal-append.queue_time_us":615177}
I20260317 16:18:33.139003 20118 raft_consensus.cc:2393] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:33.190661 21135 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.263649 21072 raft_consensus.cc:3055] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:33.262554 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:33.265511 21280 consensus_meta.cc:287] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 1.097s user 0.014s sys 0.000s
I20260317 16:18:33.247493 20041 rpcz_store.cc:276] 0317 16:18:30.309590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.551049 (+2241459us) service_pool.cc:225] Handling call
0317 16:18:32.628619 (+ 77570us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.764636 (+136017us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:32.777110 (+ 12474us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.902856 (+125746us) log.cc:844] Serialized 26 byte log entry
0317 16:18:33.057660 (+154804us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:33.061246 (+ 3586us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.074164 (+ 12918us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.115043 (+ 40879us) raft_consensus.cc:1685] finished
0317 16:18:33.115062 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.115204 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":230750,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":230,"thread_start_us":111301,"threads_started":1,"wal-append.queue_time_us":195935}
I20260317 16:18:33.267009 20265 raft_consensus.cc:2376] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:33.261118 21270 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:33.267937 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 86603.8us
I20260317 16:18:33.362123 20016 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:33.268409 21216 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:33.268784 20199 rpcz_store.cc:276] 0317 16:18:31.421921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.122307 (+700386us) service_pool.cc:225] Handling call
0317 16:18:32.122656 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.123439 (+ 783us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.123484 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.123965 (+ 481us) log.cc:844] Serialized 26 byte log entry
0317 16:18:32.456042 (+332077us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:32.456102 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.456197 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.182208 (+726011us) raft_consensus.cc:1685] finished
0317 16:18:33.182236 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.182404 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":130,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"thread_start_us":313940,"threads_started":1,"wal-append.queue_time_us":520214}
I20260317 16:18:33.321965 21156 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:33.263381 20198 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:33.265534 21327 raft_consensus.cc:697] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:33.187556 21169 consensus_meta.cc:287] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.751s user 0.008s sys 0.004s
I20260317 16:18:33.288555 21349 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:33.365275 21216 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.368937 21216 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:33.289808 20091 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:33.369784 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 80129.1us
I20260317 16:18:33.371016 20091 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.371392 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 191) took 1604 ms. Trace:
I20260317 16:18:33.293521 20050 rpcz_store.cc:276] 0317 16:18:30.003963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.387722 (+2383759us) service_pool.cc:225] Handling call
0317 16:18:32.388158 (+ 436us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.389983 (+ 1825us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.390028 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.399756 (+ 9728us) log.cc:844] Serialized 26 byte log entry
0317 16:18:32.815942 (+416186us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:33.001300 (+185358us) log.cc:844] Serialized 14 byte log entry
0317 16:18:33.001517 (+ 217us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.001603 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.114202 (+112599us) raft_consensus.cc:1685] finished
0317 16:18:33.114224 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.114407 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1234,"prepare.queue_time_us":387233,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":29447,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":18389,"thread_start_us":479535,"threads_started":2,"wal-append.queue_time_us":671951}
I20260317 16:18:33.371887 20041 rpcz_store.cc:276] 0317 16:18:31.766805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.364776 (+1597971us) service_pool.cc:225] Handling call
0317 16:18:33.371323 (+ 6547us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.298676 20165 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.304253 20318 tablet_bootstrap.cc:492] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:33.296061 21164 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:33.308632 20893 consensus_queue.cc:237] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.350183 21072 raft_consensus.cc:740] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:33.296236 20266 rpcz_store.cc:276] 0317 16:18:30.081678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.269355 (+2187677us) service_pool.cc:225] Handling call
0317 16:18:32.294075 (+ 24720us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.309747 (+ 15672us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:32.961950 (+652203us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b18003f8d40. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea38ea785c 00007fea3623539b 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea3624f3e5 00007fea3cd025f7 00007fea3930a8d7 00007fea39319fef
0317 16:18:32.962109 (+ 159us) log.cc:844] Serialized 14 byte log entry
0317 16:18:33.158960 (+196851us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:33.172184 (+ 13224us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:33.180160 (+ 7976us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.190786 (+ 10626us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.199099 (+ 8313us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8016,"prepare.queue_time_us":304007,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":11227,"raft.queue_time_us":36216,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":11116,"spinlock_wait_cycles":21540096,"thread_start_us":428600,"threads_started":2,"wal-append.queue_time_us":334526}
I20260317 16:18:33.293226 20045 rpcz_store.cc:276] 0317 16:18:29.706295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.229506 (+2523211us) service_pool.cc:225] Handling call
0317 16:18:32.230051 (+ 545us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.230557 (+ 506us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.230615 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.231283 (+ 668us) log.cc:844] Serialized 26 byte log entry
0317 16:18:32.594872 (+363589us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.597747 (+ 2875us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.613862 (+ 16115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.187304 (+573442us) raft_consensus.cc:1685] finished
0317 16:18:33.190105 (+ 2801us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.198402 (+ 8297us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":171,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":240,"thread_start_us":296583,"threads_started":1,"wal-append.queue_time_us":466809}
I20260317 16:18:33.341495 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:33.234198 20042 rpcz_store.cc:276] 0317 16:18:30.308860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.398256 (+2089396us) service_pool.cc:225] Handling call
0317 16:18:32.494342 (+ 96086us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.598580 (+104238us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.605080 (+ 6500us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.764703 (+159623us) log.cc:844] Serialized 26 byte log entry
0317 16:18:33.006460 (+241757us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:33.006527 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.006610 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.111280 (+104670us) raft_consensus.cc:1685] finished
0317 16:18:33.111303 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.111541 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2017,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":11355,"thread_start_us":209564,"threads_started":1,"wal-append.queue_time_us":323004}
I20260317 16:18:33.311254 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.372210 20091 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:33.311592 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.261013 21210 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.262965 21115 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.295133 20119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 179) took 1681 ms. Trace:
I20260317 16:18:33.283442 21329 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.283504 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.245231 21242 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:33.138433 20892 consensus_meta.cc:287] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.815s user 0.013s sys 0.003s
I20260317 16:18:33.291203 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:33.350247 20015 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:33.293824 21343 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:33.301348 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 176) took 2991 ms. Trace:
I20260317 16:18:33.300395 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 228) took 1298 ms. Trace:
W20260317 16:18:33.408151 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 58000us
I20260317 16:18:33.364077 20016 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.287146 20270 rpcz_store.cc:276] 0317 16:18:30.467078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.253825 (+2786747us) service_pool.cc:225] Handling call
0317 16:18:33.254146 (+ 321us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.255412 (+ 1266us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.255625 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.365468 21224 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:33.310613 21110 consensus_queue.cc:260] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.373183 20050 raft_consensus.cc:1275] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260317 16:18:33.376713 20048 raft_consensus.cc:1240] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:33.395090 20238 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:33.382478 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 83909.5us
I20260317 16:18:33.357949 20197 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:33.397825 20269 raft_consensus.cc:2393] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:33.461026 20165 leader_election.cc:304] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:33.365236 20196 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:33.399106 21312 raft_consensus.cc:2749] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:33.394075 20264 raft_consensus.cc:2393] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:33.373342 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.363502 21280 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:33.394521 21209 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:33.408236 21349 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260317 16:18:33.492938 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 151392us
I20260317 16:18:33.414882 21164 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:33.569471 20320 tablet_bootstrap.cc:492] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:33.570061 20320 ts_tablet_manager.cc:1403] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.542s user 0.017s sys 0.006s
I20260317 16:18:33.380585 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.394479 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 199) took 1509 ms. Trace:
I20260317 16:18:33.373380 21211 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:33.542963 20196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55456 (request call id 269) took 1129 ms. Trace:
W20260317 16:18:33.498483 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:33.405918 21262 consensus_meta.cc:287] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.936s user 0.012s sys 0.001s
W20260317 16:18:33.521273 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 60366.6us
W20260317 16:18:33.462253 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 67284.9us
I20260317 16:18:33.412271 21210 raft_consensus.cc:3060] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:33.396550 21316 log.cc:927] Time spent T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.513s user 0.006s sys 0.008s
I20260317 16:18:33.566373 21110 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:33.455050 21201 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:33.373214 20041 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 4. Current term is 5. Ops: []
I20260317 16:18:33.372969 21324 consensus_queue.cc:237] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.379873 21270 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:33.633235 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 269248us
I20260317 16:18:33.634411 20016 leader_election.cc:400] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3.
I20260317 16:18:33.634745 20016 leader_election.cc:403] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:33.567310 20265 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:33.378159 21135 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.411336 21343 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.411180 21334 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.500211 21224 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:33.411875 21329 raft_consensus.cc:3060] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:33.537335 20042 raft_consensus.cc:1240] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:33.531225 21282 raft_consensus.cc:2804] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:33.413040 21145 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.412186 20892 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.607156 20045 rpcz_store.cc:276] 0317 16:18:31.884857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.379919 (+1495062us) service_pool.cc:225] Handling call
0317 16:18:33.393690 (+ 13771us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.394153 (+ 463us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.394412 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.455416 20192 raft_consensus.cc:2468] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:33.544432 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:33.520661 20050 pending_rounds.cc:85] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Aborting all ops after (but not including) 2
I20260317 16:18:33.671243 20050 pending_rounds.cc:107] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260317 16:18:33.408646 20116 rpcz_store.cc:276] 0317 16:18:32.001877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.005898 (+ 4021us) service_pool.cc:225] Handling call
0317 16:18:33.296295 (+1290397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.380457 21169 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.616472 20091 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.488519 21219 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:33.655673 21282 raft_consensus.cc:493] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.374864 21072 consensus_queue.cc:260] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.565217 20267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 199) took 3749 ms. Trace:
I20260317 16:18:33.479840 21320 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.564656 20043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 175) took 3253 ms. Trace:
I20260317 16:18:33.396510 21337 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:33.671617 20264 raft_consensus.cc:1240] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:33.619266 20121 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:33.629195 21349 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:33.399732 21327 consensus_queue.cc:237] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.559763 20117 raft_consensus.cc:2393] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:33.640165 21201 raft_consensus.cc:697] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:33.690035 21282 raft_consensus.cc:3060] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:33.647450 21164 raft_consensus.cc:697] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:33.666736 20125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 235) took 1344 ms. Trace:
W20260317 16:18:33.751225 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.671630 20050 raft_consensus.cc:2889] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:33.673250 20320 raft_consensus.cc:359] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:33.649207 21343 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:33.620563 20165 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.658218 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 195) took 1814 ms. Trace:
I20260317 16:18:33.614586 20196 rpcz_store.cc:276] 0317 16:18:32.410046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.412238 (+ 2192us) service_pool.cc:225] Handling call
0317 16:18:33.105978 (+693740us) spinlock_profiling.cc:243] Waited 9.2 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:33.262343 (+156365us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:33.539306 (+276963us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46861312}
I20260317 16:18:33.675635 20238 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:33.685706 20162 leader_election.cc:336] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 2.095s (SENT)
W20260317 16:18:33.688239 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 53982.5us
I20260317 16:18:33.659842 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 212) took 2539 ms. Trace:
I20260317 16:18:33.662505 21325 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:33.663604 21006 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:33.688263 20273 raft_consensus.cc:1240] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:33.762027 20016 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
W20260317 16:18:33.690970 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 74632us
I20260317 16:18:33.666268 21224 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.564299 21209 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.653704 21334 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.693778 20194 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:33.699450 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:33.763809 21277 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.775578 21329 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.408416 20046 rpcz_store.cc:276] 0317 16:18:30.309389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.522061 (+2212672us) service_pool.cc:225] Handling call
0317 16:18:32.609815 (+ 87754us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.912621 (+302806us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.925914 (+ 13293us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:33.026606 (+100692us) log.cc:844] Serialized 26 byte log entry
0317 16:18:33.199975 (+173369us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:33.207856 (+ 7881us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.225277 (+ 17421us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.270852 (+ 45575us) raft_consensus.cc:1685] finished
0317 16:18:33.286504 (+ 15652us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.298153 (+ 11649us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":279106,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":360413,"threads_started":2,"wal-append.queue_time_us":229365}
I20260317 16:18:33.701733 20318 tablet_bootstrap.cc:654] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:33.669948 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.701890 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:33.696311 21320 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:33.703433 21355 log.cc:927] Time spent T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260317 16:18:33.703847 20271 raft_consensus.cc:1240] T 1ea78aa57aaf4f86b53cd07ab29faa49 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:33.693600 20043 rpcz_store.cc:276] 0317 16:18:30.309231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.424339 (+2115108us) service_pool.cc:225] Handling call
0317 16:18:32.594582 (+170243us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.877418 (+282836us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.880095 (+ 2677us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.937002 (+ 56907us) log.cc:844] Serialized 26 byte log entry
0317 16:18:33.269207 (+332205us) spinlock_profiling.cc:243] Waited 6.23 ms on lock 0x7b1800180be0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36234d17 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea36251b92 00007fea38e995ab 00007fea38e9f0c3 00007fea38e9f471 00007fea392bb93f
0317 16:18:33.366127 (+ 96920us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:33.367227 (+ 1100us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.410702 (+ 43475us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.460239 (+ 49537us) raft_consensus.cc:1685] finished
0317 16:18:33.468953 (+ 8714us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.562143 (+ 93190us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":309540,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"spinlock_wait_cycles":14322688,"thread_start_us":293404,"threads_started":2,"wal-append.queue_time_us":431840}
I20260317 16:18:33.707748 21072 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:33.710383 20197 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:33.806128 20197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 216) took 1186 ms. Trace:
I20260317 16:18:33.806463 20197 rpcz_store.cc:276] 0317 16:18:32.620121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.241903 (+621782us) service_pool.cc:225] Handling call
0317 16:18:33.806063 (+564160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.712355 20321 tablet_bootstrap.cc:492] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:33.627722 21262 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.807286 20321 ts_tablet_manager.cc:1403] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 1.781s user 0.035s sys 0.024s
I20260317 16:18:33.713054 21349 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:33.812983 20197 raft_consensus.cc:1240] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:33.716894 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.717331 21201 consensus_queue.cc:237] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.710278 21211 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.689711 21242 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:33.409765 20119 rpcz_store.cc:276] 0317 16:18:31.614052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.138192 (+524140us) service_pool.cc:225] Handling call
0317 16:18:32.138599 (+ 407us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.161053 (+ 22454us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:32.705820 (+544767us) log.cc:844] Serialized 14 byte log entry
0317 16:18:32.933809 (+227989us) log.cc:844] Serialized 14 byte log entry
0317 16:18:33.229254 (+295445us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:33.229972 (+ 718us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:33.233672 (+ 3700us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.249495 (+ 15823us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.291675 (+ 42180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37956,"prepare.queue_time_us":428625,"prepare.run_cpu_time_us":141,"prepare.run_wall_time_us":12282,"raft.queue_time_us":66587,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":163,"thread_start_us":879966,"threads_started":3,"wal-append.queue_time_us":205970}
I20260317 16:18:33.719720 21280 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:33.719828 20088 leader_election.cc:336] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.542s (SENT)
I20260317 16:18:33.812131 21262 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:33.736212 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 144) took 2143 ms. Trace:
I20260317 16:18:33.755445 20041 rpcz_store.cc:276] 0317 16:18:31.840636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.372639 (+1532003us) service_pool.cc:225] Handling call
0317 16:18:33.373014 (+ 375us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.636273 (+263259us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.654300 (+ 18027us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.693436 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:33.730734 20272 raft_consensus.cc:1240] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:33.732997 20120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 231) took 1549 ms. Trace:
I20260317 16:18:33.746376 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:33.818341 21211 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:33.746909 21210 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:33.752996 20320 raft_consensus.cc:385] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:33.835155 20320 raft_consensus.cc:740] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:33.753217 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 194) took 1943 ms. Trace:
I20260317 16:18:33.814735 21365 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:33.751811 20125 rpcz_store.cc:276] 0317 16:18:32.322059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.322234 (+ 175us) service_pool.cc:225] Handling call
0317 16:18:32.322595 (+ 361us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.323026 (+ 431us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.323069 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.323705 (+ 636us) log.cc:844] Serialized 26 byte log entry
0317 16:18:32.527933 (+204228us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:32.527990 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.528087 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.395952 (+867865us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:33.658155 (+262203us) raft_consensus.cc:1685] finished
0317 16:18:33.658189 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.666660 (+ 8471us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2378,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3213,"spinlock_wait_cycles":43186944,"thread_start_us":203838,"threads_started":1,"wal-append.queue_time_us":461940}
W20260317 16:18:33.756678 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 136193us
I20260317 16:18:33.837822 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 254) took 2219 ms. Trace:
W20260317 16:18:33.730679 20269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 143) took 2612 ms (client timeout 2146 ms). Trace:
I20260317 16:18:33.760830 21157 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 16:18:33.761296 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 85809us
I20260317 16:18:33.836043 20050 rpcz_store.cc:276] 0317 16:18:31.810150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.372531 (+1562381us) service_pool.cc:225] Handling call
0317 16:18:33.372857 (+ 326us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.752954 (+380097us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.753165 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.840029 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:33.761870 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 76198.8us
W20260317 16:18:33.765002 20236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.840776 20050 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:33.838245 20049 rpcz_store.cc:276] 0317 16:18:31.618962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.275558 (+1656596us) service_pool.cc:225] Handling call
0317 16:18:33.303791 (+ 28233us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:33.304252 (+ 461us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:33.304290 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:33.316479 (+ 12189us) log.cc:844] Serialized 26 byte log entry
0317 16:18:33.635538 (+319059us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:33.659915 (+ 24377us) log.cc:844] Serialized 14 byte log entry
0317 16:18:33.660146 (+ 231us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.660221 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.837581 (+177360us) raft_consensus.cc:1685] finished
0317 16:18:33.837608 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.837765 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8674,"prepare.queue_time_us":23821,"prepare.run_wall_time_us":4306,"raft.queue_time_us":11465,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":12707,"spinlock_wait_cycles":17039744,"thread_start_us":267544,"threads_started":1,"wal-append.queue_time_us":298466}
I20260317 16:18:33.773260 21337 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:33.763509 21156 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.844275 21156 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:33.700820 20892 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:33.722739 20191 raft_consensus.cc:1275] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:33.731422 21225 raft_consensus.cc:3055] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:33.728832 21154 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:33.845273 21225 raft_consensus.cc:740] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:33.846154 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:33.846426 21225 consensus_queue.cc:260] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:33.847532 21225 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:33.847950 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:33.848795 20191 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:33.767117 21333 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 16:18:33.838641 20269 rpcz_store.cc:269] 0317 16:18:31.118440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:31.917566 (+799126us) service_pool.cc:225] Handling call
0317 16:18:33.730285 (+1812719us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400225e54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:33.730573 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3324074240}
W20260317 16:18:33.780236 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:33.856848 20239 leader_election.cc:343] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:33.763294 21352 log.cc:927] Time spent T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
W20260317 16:18:33.760682 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.771090 21362 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.858351 21362 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.771996 21321 raft_consensus.cc:2804] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:33.860795 21321 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:33.777596 21006 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:33.861132 20191 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:33.862347 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:33.665890 21145 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:33.822433 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 102641us
W20260317 16:18:33.868412 20088 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.868225 20091 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:33.768460 20270 rpcz_store.cc:276] 0317 16:18:31.117188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.463648 (+2346460us) service_pool.cc:225] Handling call
0317 16:18:33.610434 (+146786us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.642151 (+ 31717us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.656232 (+ 14081us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:33.779541 21224 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.826992 20321 raft_consensus.cc:359] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.870216 20321 raft_consensus.cc:385] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:33.690651 20267 rpcz_store.cc:276] 0317 16:18:29.816393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.167471 (+2351078us) service_pool.cc:225] Handling call
0317 16:18:32.168358 (+ 887us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.614740 (+446382us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.530608 (+915868us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400225954. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:33.563106 (+ 32498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":691075968}
I20260317 16:18:33.799088 20190 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:33.862687 21006 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.829766 20266 raft_consensus.cc:1240] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20260317 16:18:33.841765 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.818557 21135 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.870275 20091 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.870554 20321 raft_consensus.cc:740] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:33.825202 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:33.878543 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:33.872328 21333 raft_consensus.cc:2804] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:33.879483 21135 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:33.881203 20193 raft_consensus.cc:1240] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:33.881374 21210 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:33.881534 21135 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:33.882268 21310 log.cc:927] Time spent T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 1.022s user 0.001s sys 0.015s
I20260317 16:18:33.874162 20321 consensus_queue.cc:260] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.861198 21321 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:33.870641 21365 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:33.884172 20050 raft_consensus.cc:1275] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:33.879881 21333 raft_consensus.cc:697] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:33.863415 21209 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:33.825489 21216 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:33.896399 20238 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:33.903002 20165 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:33.905619 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:33.885730 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
@ 0x7fea32b067f1 abort at ??:0
I20260317 16:18:33.894603 20044 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:33.888267 20043 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:33.938930 21362 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.939111 21329 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:33.986043 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:33.988003 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:33.989024 20043 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:33.939266 21366 consensus_queue.cc:1048] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260317 16:18:33.990048 20088 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:33.940114 21334 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:33.992818 21334 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:33.940178 21037 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.940958 21169 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.942605 20268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 202) took 4116 ms. Trace:
I20260317 16:18:33.828011 20264 rpcz_store.cc:276] 0317 16:18:31.593027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.625237 (+2032210us) service_pool.cc:225] Handling call
0317 16:18:33.656542 (+ 31305us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.711757 (+ 55215us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.736145 (+ 24388us) inbound_call.cc:173] Queueing success response
Metrics: {}
@ 0x2d3616 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260317 16:18:33.896106 21210 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:33.948446 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:33.892346 21006 consensus_queue.cc:1048] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260317 16:18:33.873024 21349 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:33.939114 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.954933 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:33.946156 21145 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:33.946066 20321 ts_tablet_manager.cc:1434] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.138s user 0.006s sys 0.000s
W20260317 16:18:33.967101 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 61582.3us
W20260317 16:18:33.967705 20236 leader_election.cc:343] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:33.980897 20265 raft_consensus.cc:1275] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260317 16:18:33.995077 20041 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:33.818866 21164 consensus_queue.cc:237] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:33.981531 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 79032.8us
I20260317 16:18:33.891906 21154 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:33.990171 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:33.894618 20892 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:33.834600 20120 rpcz_store.cc:276] 0317 16:18:32.183936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.241988 (+ 58052us) service_pool.cc:225] Handling call
0317 16:18:32.242370 (+ 382us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:32.311498 (+ 69128us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:32.659640 (+348142us) log.cc:844] Serialized 14 byte log entry
0317 16:18:32.969636 (+309996us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:32.981171 (+ 11535us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:33.006082 (+ 24911us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:33.732716 (+726634us) spinlock_profiling.cc:243] Waited 56.9 ms on lock 0x7b54000acb54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:33.732774 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.732931 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":312279,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":94754,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":130841984,"thread_start_us":455637,"threads_started":2,"wal-append.queue_time_us":516725}
I20260317 16:18:33.997133 20162 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:34.043473 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 46439.1us
I20260317 16:18:34.004318 20042 raft_consensus.cc:1240] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:34.004634 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:33.997601 20268 rpcz_store.cc:276] 0317 16:18:29.825811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.167785 (+2341974us) service_pool.cc:225] Handling call
0317 16:18:32.168201 (+ 416us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:32.168697 (+ 496us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:32.168732 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:32.169265 (+ 533us) log.cc:844] Serialized 26 byte log entry
0317 16:18:32.601636 (+432371us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:32.612771 (+ 11135us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:32.620887 (+ 8116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:33.882002 (+1261115us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b2400010e98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea360a6b34 00007fea360a4c3b 00007fea36133b39 00007fea361337c5 00007fea36132e0d 00007fea38e834ae 00007fea38e9ac2c 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1
0317 16:18:33.942142 (+ 60140us) raft_consensus.cc:1685] finished
0317 16:18:33.942193 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:33.942514 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2290,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":55158528,"thread_start_us":381820,"threads_started":1,"wal-append.queue_time_us":574233}
I20260317 16:18:34.009016 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:33.986093 21329 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:33.998071 21235 consensus_queue.cc:1048] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260317 16:18:33.981385 21156 raft_consensus.cc:2804] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:33.994428 21277 consensus_queue.cc:1059] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260317 16:18:34.048866 20193 raft_consensus.cc:1240] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:33.980522 21209 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:33.995533 21373 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:34.056512 20238 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:34.049307 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.038095 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.037564 21219 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.077898 21219 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:34.080155 21365 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.030015 20236 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:34.042578 21154 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.061797 21208 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:34.043865 21006 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.036700 21349 consensus_queue.cc:1048] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260317 16:18:34.043126 20269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 217) took 1310 ms. Trace:
I20260317 16:18:34.047626 21312 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:34.052191 20892 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.072526 21209 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:34.082726 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 115070us
I20260317 16:18:34.027456 21210 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.090078 21006 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.087394 21145 raft_consensus.cc:2749] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:34.082579 21156 raft_consensus.cc:697] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:34.095085 21224 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.095901 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.097098 20318 tablet_bootstrap.cc:492] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:34.135675 20318 ts_tablet_manager.cc:1403] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 0.844s user 0.010s sys 0.030s
I20260317 16:18:34.099372 20270 raft_consensus.cc:1240] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:34.048573 21157 consensus_queue.cc:1059] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260317 16:18:34.083936 21320 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.069824 21329 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:34.100786 21262 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:34.044306 21333 consensus_queue.cc:237] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.108873 21343 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:34.111426 21219 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.071494 21280 consensus_queue.cc:1059] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260317 16:18:34.103888 21208 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.120234 21334 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:34.121528 21340 raft_consensus.cc:3055] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:34.126199 21325 raft_consensus.cc:2749] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 2. Current term is 3.
I20260317 16:18:34.130311 21211 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.115571 21209 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.152446 21006 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.152452 21208 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:34.142650 21216 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.159605 21340 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:34.153748 21321 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.163559 21282 raft_consensus.cc:515] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.174415 21343 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:34.174962 21282 leader_election.cc:290] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:34.174932 21321 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.175843 21329 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:34.037014 20016 connection.cc:232] RPC call timeout handler was delayed by -1.7290120530001332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
I20260317 16:18:33.978811 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.047873 20043 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:34.028796 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.040542 20090 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:34.229056 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:34.229979 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:34.231024 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 190408us
I20260317 16:18:34.235385 20015 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:34.071082 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.237515 20196 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:34.240409 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.241767 20049 raft_consensus.cc:2376] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:34.242233 20196 raft_consensus.cc:1275] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:34.243172 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:34.244182 20049 raft_consensus.cc:3060] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:34.244936 20015 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:34.067023 20191 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
W20260317 16:18:34.113336 20236 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.791s (SENT) [suppressed 31 similar messages]
I20260317 16:18:34.076670 20264 raft_consensus.cc:2393] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:34.099635 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 216) took 1392 ms. Trace:
I20260317 16:18:34.309697 21224 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:34.310236 21224 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:34.109462 20267 raft_consensus.cc:2393] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:34.312749 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.313218 21169 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:34.313696 20267 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:34.105988 20269 rpcz_store.cc:276] 0317 16:18:32.732608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.851770 (+1119162us) service_pool.cc:225] Handling call
0317 16:18:33.852154 (+ 384us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.852367 (+ 213us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:33.852797 (+ 430us) log.cc:844] Serialized 14 byte log entry
0317 16:18:33.999951 (+147154us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:34.019076 (+ 19125us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:34.025352 (+ 6276us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.032916 (+ 7564us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:34.040039 (+ 7123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52782,"prepare.queue_time_us":256,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":93542,"spinlock_wait_cycles":8235392,"thread_start_us":42132,"threads_started":1,"wal-append.queue_time_us":181067}
I20260317 16:18:34.314106 21169 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:34.316315 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.317129 20269 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.317174 21169 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.137475 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 213) took 2888 ms. Trace:
I20260317 16:18:34.319180 20271 rpcz_store.cc:276] 0317 16:18:31.249211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.569051 (+2319840us) service_pool.cc:225] Handling call
0317 16:18:33.627110 (+ 58059us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.810451 (+183341us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.136927 (+326476us) spinlock_profiling.cc:243] Waited 56.6 ms on lock 0x7b5400418a54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.137318 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130276608}
I20260317 16:18:34.321252 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.267815 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 151) took 1177 ms. Trace:
I20260317 16:18:34.322924 20270 rpcz_store.cc:276] 0317 16:18:33.089867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.052009 (+962142us) service_pool.cc:225] Handling call
0317 16:18:34.099139 (+ 47130us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:34.267104 (+167965us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.267579 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:34.324347 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.324730 20892 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.325181 20270 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:34.134475 20320 consensus_queue.cc:260] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.176575 21340 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.276090 21154 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:34.327081 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.139824 21156 consensus_queue.cc:237] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.327955 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.328184 20264 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.329166 20320 ts_tablet_manager.cc:1434] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.759s user 0.006s sys 0.005s
W20260317 16:18:34.329572 20320 ts_tablet_manager.cc:1467] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Tablet startup took 759ms
I20260317 16:18:34.277529 21210 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.147770 21262 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.326547 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.292263 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.296037 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.296013 21329 raft_consensus.cc:2804] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:34.310680 21224 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:34.308823 21164 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:34.309015 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 195577us
I20260317 16:18:34.310138 20265 rpcz_store.cc:276] 0317 16:18:32.706732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.704048 (+997316us) service_pool.cc:225] Handling call
0317 16:18:33.704402 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:34.033601 (+329199us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.099302 (+ 65701us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54001cc5d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.099526 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":420848512}
I20260317 16:18:34.291749 20321 tablet_bootstrap.cc:492] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:34.320147 20118 raft_consensus.cc:1240] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: []
W20260317 16:18:34.335525 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:34.309715 20045 raft_consensus.cc:2376] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3. Re-sending same reply.
W20260317 16:18:34.338868 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:34.339296 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.340022 21282 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.343502 20123 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 188) took 1425 ms. Trace:
I20260317 16:18:34.344887 20046 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:34.354395 20268 raft_consensus.cc:2393] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:34.355043 21164 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.355921 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.357046 20268 raft_consensus.cc:2368] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:34.357928 21325 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:34.358981 20193 raft_consensus.cc:1275] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260317 16:18:34.359381 20193 pending_rounds.cc:85] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting all ops after (but not including) 2
I20260317 16:18:34.359800 20193 pending_rounds.cc:107] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260317 16:18:34.360129 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 238) took 4050 ms. Trace:
I20260317 16:18:34.360721 20048 rpcz_store.cc:276] 0317 16:18:30.309756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.737316 (+2427560us) service_pool.cc:225] Handling call
0317 16:18:32.792402 (+ 55086us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.486006 (+693604us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.359470 (+873464us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b5400754254. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.359919 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1322394496}
I20260317 16:18:34.362568 20116 raft_consensus.cc:2393] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:34.364907 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.366158 20271 raft_consensus.cc:2393] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:34.358239 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.360154 20193 raft_consensus.cc:2889] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:34.375599 21262 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.376092 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.375830 21344 consensus_queue.cc:1048] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:18:34.375602 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:34.373997 21329 raft_consensus.cc:697] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:34.329917 20320 ts_tablet_manager.cc:1469] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Trace:
0317 16:18:04.950380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:04.950548 (+ 168us) service_pool.cc:225] Handling call
0317 16:18:04.952727 (+ 2179us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:04.970421 (+ 17694us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.268735 (+298314us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.340379 (+ 71644us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.352856 (+ 12477us) inbound_call.cc:173] Queueing success response
0317 16:18:32.976562 (+27623706us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:33.570408 (+593846us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:33.655065 (+ 84657us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:33.655613 (+ 548us) tablet_replica.cc:280] Starting consensus
0317 16:18:34.134057 (+478444us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b4c005f3ec0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea375d7e50 00007fea375e8ca2 00007fea375af055 00007fea375d635d 00007fea375d5fc3 00007fea392fb0bb 00007fea392fbbe5 00007fea392f6e60 00007fea392f5b70 00007fea3ccfcd5e 00007fea3e30f02a
Metrics: {"raft.queue_time_us":423,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"spinlock_wait_cycles":41950080,"tablet-open.queue_time_us":27612469}
I20260317 16:18:34.388918 21277 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.389582 20318 raft_consensus.cc:359] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.389828 21277 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.390498 20318 raft_consensus.cc:385] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:34.390930 20318 raft_consensus.cc:740] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:34.391278 20236 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:34.394222 21282 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.395756 20125 raft_consensus.cc:1275] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260317 16:18:34.397389 21222 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:34.397969 21363 consensus_queue.cc:1048] T b7f7f224bb064944a05175b00d835fce P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:34.382956 20271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 156) took 1077 ms. Trace:
I20260317 16:18:34.399600 20271 rpcz_store.cc:276] 0317 16:18:33.302343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.376941 (+1074598us) service_pool.cc:225] Handling call
0317 16:18:34.377564 (+ 623us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:34.377939 (+ 375us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:34.377975 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:34.378052 (+ 77us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:34.378099 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.378173 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:34.379823 (+ 1650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:34.391788 20318 consensus_queue.cc:260] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.403606 20320 tablet_bootstrap.cc:492] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
I20260317 16:18:34.404655 21277 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:34.394662 20198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55464 (request call id 220) took 1695 ms (client timeout 1579 ms). Trace:
W20260317 16:18:34.405650 20198 rpcz_store.cc:269] 0317 16:18:32.699612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.004975 (+305363us) service_pool.cc:225] Handling call
0317 16:18:34.394231 (+1389256us) spinlock_profiling.cc:243] Waited 72.8 ms on lock 0x7b54003f4dd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:34.394452 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":167349632}
I20260317 16:18:34.407109 20044 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 177) took 3876 ms. Trace:
I20260317 16:18:34.407558 20044 rpcz_store.cc:276] 0317 16:18:30.530953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.106507 (+2575554us) service_pool.cc:225] Handling call
0317 16:18:33.140405 (+ 33898us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.984838 (+844433us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.406498 (+421660us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54002440d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.406892 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1684711296}
I20260317 16:18:34.408675 21242 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.408599 21329 consensus_queue.cc:237] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.409389 21242 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.401800 20271 raft_consensus.cc:1240] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 3. Ops: []
I20260317 16:18:34.420658 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:34.421393 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:34.421859 20199 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:34.348857 20123 rpcz_store.cc:276] 0317 16:18:32.918395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.021209 (+1102814us) service_pool.cc:225] Handling call
0317 16:18:34.039526 (+ 18317us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:34.059599 (+ 20073us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:34.233025 (+173426us) log.cc:844] Serialized 14 byte log entry
0317 16:18:34.340583 (+107558us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:34.342826 (+ 2243us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:34.342951 (+ 125us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.343085 (+ 134us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:34.343299 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4757,"prepare.queue_time_us":68795,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":57,"raft.queue_time_us":41655,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":37758,"thread_start_us":19302,"threads_started":1,"wal-append.queue_time_us":87478}
I20260317 16:18:34.418537 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.425935 20046 raft_consensus.cc:2468] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:34.429165 21164 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.431188 21340 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.432520 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:34.433869 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.435195 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:34.435683 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:34.442793 21325 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:34.443389 21325 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:34.444222 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.445089 20046 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:34.445161 21361 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.446116 21361 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.437361 21211 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.447989 20041 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:34.448894 21361 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.449189 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.449991 20041 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:34.422341 20045 raft_consensus.cc:2376] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5. Re-sending same reply.
I20260317 16:18:34.452262 21242 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.452811 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.454301 20045 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:34.459106 21216 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:34.460114 21333 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.465173 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 138) took 4275 ms. Trace:
I20260317 16:18:34.473429 21282 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.475297 20199 raft_consensus.cc:2468] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
W20260317 16:18:34.476608 20194 consensus_meta.cc:287] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.693s user 0.009s sys 0.009s
I20260317 16:18:34.477079 20194 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:34.477780 21115 raft_consensus.cc:2749] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:34.478950 20194 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55456 (request call id 270) took 2068 ms (client timeout 1789 ms). Trace:
W20260317 16:18:34.479359 20194 rpcz_store.cc:269] 0317 16:18:32.410636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.420355 (+ 9719us) service_pool.cc:225] Handling call
0317 16:18:33.686550 (+1266195us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b5400212928. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0b5b44 00007fea3e2c02e7 00007fea3e2bfb78 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786
0317 16:18:34.478854 (+792304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32674176}
W20260317 16:18:34.480392 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 309392us
I20260317 16:18:34.483212 20162 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:34.482987 21390 consensus_queue.cc:1059] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 16:18:34.484298 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 475307us
I20260317 16:18:34.487455 21277 raft_consensus.cc:3055] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:34.488139 21277 raft_consensus.cc:740] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:34.489169 20239 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:34.490358 21242 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:34.489157 21277 consensus_queue.cc:260] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.461006 21333 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.495721 20043 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:34.496344 21333 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.496826 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.497188 21037 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.501017 21037 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.501760 20047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 179) took 3248 ms. Trace:
W20260317 16:18:34.499306 20088 consensus_peers.cc:597] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f -> Peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Couldn't send request to peer 9a3afb098d6a45f0b405566771a1157b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260317 16:18:34.503219 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 475608us
W20260317 16:18:34.470427 20238 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:34.510337 20195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 274) took 1729 ms. Trace:
I20260317 16:18:34.515170 21282 raft_consensus.cc:2764] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260317 16:18:34.517794 20195 rpcz_store.cc:276] 0317 16:18:32.781335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.506326 (+1724991us) service_pool.cc:225] Handling call
0317 16:18:34.509481 (+ 3155us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:34.509824 (+ 343us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:34.509863 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:34.509915 (+ 52us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:34.509996 (+ 81us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.510067 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:34.510165 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:34.519361 20162 leader_election.cc:343] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:34.510869 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.502297 20047 rpcz_store.cc:276] 0317 16:18:31.253382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.907577 (+1654195us) service_pool.cc:225] Handling call
0317 16:18:33.003803 (+ 96226us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.247022 (+243219us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.501134 (+1254112us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540049cad4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.501461 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407732480}
I20260317 16:18:34.520612 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46060 (request call id 194) took 1242 ms. Trace:
I20260317 16:18:34.523255 20016 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:34.517509 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.525218 20042 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
W20260317 16:18:34.526216 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 49127us
I20260317 16:18:34.541882 20264 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:34.491554 20273 rpcz_store.cc:276] 0317 16:18:30.189932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.603306 (+2413374us) service_pool.cc:225] Handling call
0317 16:18:32.636415 (+ 33109us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.768138 (+1131723us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.464629 (+696491us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400226354. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.464969 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2418630016}
I20260317 16:18:34.530510 21312 raft_consensus.cc:3055] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:34.521692 20162 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:34.521641 20121 rpcz_store.cc:276] 0317 16:18:33.278092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.283653 (+ 5561us) service_pool.cc:225] Handling call
0317 16:18:34.519828 (+1236175us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54002e1654. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:34.520025 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":454342912}
I20260317 16:18:34.533209 20320 tablet_bootstrap.cc:654] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:34.577435 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.534701 21037 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.515451 20122 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260317 16:18:34.585480 20122 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46060 (request call id 193) took 1307 ms. Trace:
I20260317 16:18:34.533043 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34240 (request call id 175) took 3567 ms. Trace:
I20260317 16:18:34.535844 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.588012 20048 raft_consensus.cc:2393] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:34.587352 20266 rpcz_store.cc:276] 0317 16:18:31.028324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.376202 (+2347878us) service_pool.cc:225] Handling call
0317 16:18:33.376692 (+ 490us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:33.879067 (+502375us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.532403 (+653336us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54004d9c54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.532860 (+ 457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1001835904}
I20260317 16:18:34.585938 20122 rpcz_store.cc:276] 0317 16:18:33.277622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.283572 (+ 5950us) service_pool.cc:225] Handling call
0317 16:18:34.514686 (+1231114us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:34.514980 (+ 294us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:34.585072 (+ 70092us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.585365 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47878912}
I20260317 16:18:34.589130 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.537402 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.523180 21209 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.510720 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:34.592581 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:34.517071 20088 leader_election.cc:336] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 2.093s (SENT)
I20260317 16:18:34.526964 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:34.542591 21164 raft_consensus.cc:2749] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:34.522185 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:34.543699 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 205732us
I20260317 16:18:34.532963 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 209) took 4068 ms. Trace:
I20260317 16:18:34.544313 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.546607 21115 consensus_queue.cc:1059] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:34.547286 21390 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:34.446183 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:34.615489 20088 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:34.562631 20043 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:34.563392 20318 ts_tablet_manager.cc:1434] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.427s user 0.006s sys 0.006s
W20260317 16:18:34.639796 20318 ts_tablet_manager.cc:1467] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Tablet startup took 503ms
I20260317 16:18:34.564010 21224 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.564441 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:34.565389 21277 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:34.510495 20267 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
W20260317 16:18:34.565860 19945 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46692 (request call id 39) took 13588 ms (client timeout 15000 ms). Trace:
I20260317 16:18:34.567610 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:34.456518 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.568881 21345 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.570129 20321 tablet_bootstrap.cc:654] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:34.534119 20117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 239) took 1248 ms. Trace:
I20260317 16:18:34.578629 21312 raft_consensus.cc:740] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:34.592514 20048 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
W20260317 16:18:34.600643 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 44954.2us
I20260317 16:18:34.599243 20049 raft_consensus.cc:2468] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:34.641577 20238 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:34.600004 20044 raft_consensus.cc:2468] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:34.648893 20199 raft_consensus.cc:2393] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:34.610589 20272 rpcz_store.cc:276] 0317 16:18:30.464123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:32.648309 (+2184186us) service_pool.cc:225] Handling call
0317 16:18:33.730501 (+1082192us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.835135 (+104634us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.532437 (+697302us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400225e50. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:34.532652 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2372181888}
I20260317 16:18:34.577181 21325 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:34.640167 20318 ts_tablet_manager.cc:1469] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Trace:
0317 16:18:05.292902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:05.334875 (+ 41973us) service_pool.cc:225] Handling call
0317 16:18:05.337533 (+ 2658us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 16:18:05.337828 (+ 295us) ts_tablet_manager.cc:682] Creating new metadata...
0317 16:18:05.363148 (+ 25320us) tablet_metadata.cc:641] Metadata flushed
0317 16:18:05.402022 (+ 38874us) tablet_replica.cc:206] Creating consensus instance
0317 16:18:05.405056 (+ 3034us) inbound_call.cc:173] Queueing success response
0317 16:18:33.270550 (+27865494us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 16:18:34.136072 (+865522us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 16:18:34.289063 (+152991us) tablet_replica.cc:261] Starting instrumentation
0317 16:18:34.290199 (+ 1136us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":29737,"raft.queue_time_us":55061,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":285,"tablet-open.queue_time_us":27865589}
I20260317 16:18:34.662904 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.609422 21222 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.697631 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:34.702843 20198 raft_consensus.cc:1275] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260317 16:18:34.703382 20198 pending_rounds.cc:85] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting all ops after (but not including) 1
I20260317 16:18:34.703663 20198 pending_rounds.cc:107] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260317 16:18:34.659904 20041 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.708954 20197 raft_consensus.cc:2393] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:34.709926 20198 raft_consensus.cc:2889] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260317 16:18:34.706184 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 53764.5us
I20260317 16:18:34.693444 21363 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:34.717140 20050 raft_consensus.cc:2368] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4.
I20260317 16:18:34.662638 21164 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:34.649380 20046 raft_consensus.cc:2376] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:34.646831 20269 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:34.705428 20270 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:34.719924 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:34.735561 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 69213us
I20260317 16:18:34.738091 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.732484 20045 raft_consensus.cc:2393] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:34.739966 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 161109us
I20260317 16:18:34.709688 21154 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:34.745564 20197 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:34.745527 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:34.737159 21006 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:34.747403 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.752902 20195 raft_consensus.cc:2468] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:34.741236 20049 raft_consensus.cc:1275] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:34.732542 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.739647 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:34.756526 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 235015us
I20260317 16:18:34.756860 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.699610 21345 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.781891 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.768630 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.774047 21115 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.737989 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
@ 0x7fea358dada4 google::LogMessage::SendToLog() at ??:0
I20260317 16:18:34.801242 21224 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:34.802388 20318 tablet_bootstrap.cc:492] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f: Bootstrap starting.
I20260317 16:18:34.803131 20165 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:34.805135 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:34.768455 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.792095 20044 raft_consensus.cc:1240] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:34.811755 20122 raft_consensus.cc:2410] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:34.811910 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.796859 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.813294 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.813687 21362 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:34.814568 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.814332 21362 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.815232 20273 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.816661 21362 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.809827 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.817869 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.802300 21115 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.819329 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.742235 21222 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:34.791638 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 274603us
I20260317 16:18:34.793757 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.781383 21342 raft_consensus.cc:2804] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:34.662281 20117 rpcz_store.cc:276] 0317 16:18:33.285945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:33.715697 (+429752us) service_pool.cc:225] Handling call
0317 16:18:33.716154 (+ 457us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:33.818997 (+102843us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:33.821169 (+ 2172us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:33.828433 (+ 7264us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:33.829540 (+ 1107us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.533642 (+704102us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b540016dd54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:34.533716 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:34.533935 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":222196608}
I20260317 16:18:34.762598 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.780616 20015 leader_election.cc:304] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:34.798430 20123 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 5
I20260317 16:18:34.760180 20267 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20260317 16:18:34.797946 21391 log.cc:927] Time spent T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260317 16:18:34.813359 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:34.820559 21325 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.784353 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.812937 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.775678 20270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34248 (request call id 157) took 1473 ms. Trace:
I20260317 16:18:34.764389 21210 raft_consensus.cc:3055] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Stepping down as leader of term 3
W20260317 16:18:34.821213 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 44542.3us
I20260317 16:18:34.755268 21329 raft_consensus.cc:493] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:34.874065 21342 raft_consensus.cc:697] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:34.876734 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.808044 21345 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:34.878448 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 100091us
I20260317 16:18:34.890975 21224 raft_consensus.cc:2804] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:34.899814 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 94786.2us
I20260317 16:18:34.900352 21224 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:34.900816 21224 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:34.900979 21211 consensus_queue.cc:1048] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:34.810156 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.810367 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:34.821724 20118 raft_consensus.cc:2368] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:34.786732 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.913432 20197 raft_consensus.cc:2393] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:34.821480 21115 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.818684 20196 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:34.918885 21211 consensus_queue.cc:1059] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:34.878703 20043 raft_consensus.cc:1275] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260317 16:18:34.912755 20122 raft_consensus.cc:2376] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1. Re-sending same reply.
I20260317 16:18:34.755498 21164 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:34.909955 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.934005 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:34.970717 20194 raft_consensus.cc:2393] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:34.773880 21312 consensus_queue.cc:260] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:34.971714 21312 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:34.972400 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.973239 20118 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:34.938649 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:34.974289 21280 consensus_queue.cc:1175] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9a3afb098d6a45f0b405566771a1157b" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260317 16:18:34.975692 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.928097 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.976689 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:34.940071 21342 consensus_queue.cc:237] T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.941286 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:34.885618 21390 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260317 16:18:34.885044 21154 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:34.889878 21210 raft_consensus.cc:740] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:34.957715 20273 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:34.983278 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:34.971851 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:34.985814 20190 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
W20260317 16:18:34.987380 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 65269.3us
I20260317 16:18:34.898152 20042 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
W20260317 16:18:34.657681 19945 rpcz_store.cc:269] 0317 16:18:20.977700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:20.977961 (+ 261us) service_pool.cc:225] Handling call
0317 16:18:34.565568 (+13587607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:29.755110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:29.759420 (+ 4310us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:29.759477 (+ 57us) write_op.cc:435] Acquired schema lock
0317 16:18:29.759494 (+ 17us) tablet.cc:661] Decoding operations
0317 16:18:29.780452 (+ 20958us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:29.780540 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:29.780572 (+ 32us) tablet.cc:684] Acquiring locks for 8 operations
0317 16:18:29.781312 (+ 740us) tablet.cc:700] Row locks acquired
0317 16:18:29.781329 (+ 17us) write_op.cc:260] PREPARE: finished
0317 16:18:29.781506 (+ 177us) write_op.cc:270] Start()
0317 16:18:29.781627 (+ 121us) write_op.cc:276] Timestamp: P: 1773764309781443 usec, L: 0
0317 16:18:29.781646 (+ 19us) op_driver.cc:348] REPLICATION: starting
0317 16:18:29.814216 (+ 32570us) log.cc:844] Serialized 3281 byte log entry
0317 16:18:30.268060 (+453844us) op_driver.cc:464] REPLICATION: finished
0317 16:18:30.412872 (+144812us) write_op.cc:301] APPLY: starting
0317 16:18:30.412986 (+ 114us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:30.413261 (+ 275us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:30.413400 (+ 139us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:30.419516 (+ 6116us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:30.420336 (+ 820us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0317 16:18:30.420421 (+ 85us) write_op.cc:312] APPLY: finished
0317 16:18:30.420916 (+ 495us) log.cc:844] Serialized 64 byte log entry
0317 16:18:30.422956 (+ 2040us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:30.437609 (+ 14653us) write_op.cc:454] Released schema lock
0317 16:18:30.439572 (+ 1963us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:30.497947 (+ 58375us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:30.509460 (+ 11513us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":100666,"apply.run_cpu_time_us":4700,"apply.run_wall_time_us":45937,"num_ops":8,"prepare.queue_time_us":374426,"prepare.run_cpu_time_us":16925,"prepare.run_wall_time_us":185832,"raft.queue_time_us":175549,"raft.run_cpu_time_us":16127,"raft.run_wall_time_us":179725,"replication_time_us":458877,"thread_start_us":277671,"threads_started":4,"wal-append.queue_time_us":151996,"wal-append.run_cpu_time_us":23822,"wal-append.run_wall_time_us":1437354}]]}
I20260317 16:18:34.961961 21164 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:34.962697 21333 consensus_queue.cc:1048] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:34.965313 20191 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:34.966023 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 224) took 1342 ms. Trace:
I20260317 16:18:35.019188 20191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34608 (request call id 188) took 1068 ms. Trace:
I20260317 16:18:34.966156 21235 consensus_queue.cc:1048] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260317 16:18:35.019650 20191 rpcz_store.cc:276] 0317 16:18:33.951282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.008636 (+ 57354us) service_pool.cc:225] Handling call
0317 16:18:35.019090 (+1010454us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:34.966250 20236 leader_election.cc:336] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.730s (SENT)
I20260317 16:18:34.967832 21211 consensus_queue.cc:1243] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer 9a3afb098d6a45f0b405566771a1157b log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260317 16:18:34.969321 21222 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:35.021517 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 55330.8us
I20260317 16:18:34.969631 20318 tablet_bootstrap.cc:654] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:34.940332 20124 raft_consensus.cc:3060] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:35.019105 20265 rpcz_store.cc:276] 0317 16:18:33.623359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.388813 (+765454us) service_pool.cc:225] Handling call
0317 16:18:34.389343 (+ 530us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:34.389943 (+ 600us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:34.389976 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:34.390591 (+ 615us) log.cc:844] Serialized 26 byte log entry
0317 16:18:34.657304 (+266713us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:34.657384 (+ 80us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.688324 (+ 30940us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:34.965738 (+277414us) raft_consensus.cc:1685] finished
0317 16:18:34.965766 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:34.965940 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7391,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":260,"thread_start_us":225741,"threads_started":1,"wal-append.queue_time_us":551149}
I20260317 16:18:34.897528 20271 raft_consensus.cc:2468] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:34.914227 21340 consensus_queue.cc:1059] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260317 16:18:34.922623 21325 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260317 16:18:34.819376 21209 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:34.981876 20266 raft_consensus.cc:2393] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:34.974320 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:35.034356 20118 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:35.036401 20267 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260317 16:18:34.992424 20193 raft_consensus.cc:1275] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:35.039067 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:35.040035 20119 raft_consensus.cc:3060] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:35.006871 20120 raft_consensus.cc:2468] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:35.006946 21390 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.042438 21390 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:35.008052 21210 consensus_queue.cc:260] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:34.998087 21006 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:35.044741 21210 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:35.044900 21006 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:34.877986 20270 rpcz_store.cc:276] 0317 16:18:33.302712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.374598 (+1071886us) service_pool.cc:225] Handling call
0317 16:18:34.662085 (+287487us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:34.743551 (+ 81466us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:34.775372 (+ 31821us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b5400684cd0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:34.775575 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":650478464}
I20260317 16:18:35.009413 21363 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:35.047215 21363 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:35.012431 21362 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:35.048558 21362 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:35.048686 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.049073 21362 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:35.050174 21333 raft_consensus.cc:3055] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:35.050567 21333 raft_consensus.cc:740] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:35.016397 20238 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:35.051580 21333 consensus_queue.cc:260] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.053207 21345 raft_consensus.cc:2804] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:35.053629 21345 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:35.053972 21345 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:35.012035 20016 leader_election.cc:400] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:35.054939 20016 leader_election.cc:403] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260317 16:18:34.925017 21329 raft_consensus.cc:515] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:35.055773 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 43976.6us
I20260317 16:18:35.052810 21333 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:35.057868 20016 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:35.056744 21235 consensus_queue.cc:1048] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:35.057503 21211 consensus_queue.cc:1059] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260317 16:18:35.060525 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:35.063813 21329 leader_election.cc:290] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:35.064710 21329 raft_consensus.cc:3055] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:35.065140 21329 raft_consensus.cc:740] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:35.066274 21329 consensus_queue.cc:260] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:35.068300 20041 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:35.069104 21325 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:35.071712 21277 raft_consensus.cc:3055] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:35.021548 19946 catalog_manager.cc:5671] T a663a988cc754c24b227bb9bc85540db P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 2 to 4, leader changed from 9a3afb098d6a45f0b405566771a1157b (127.16.123.193) to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 4 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:35.077800 19946 catalog_manager.cc:5671] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:35.080381 19946 catalog_manager.cc:5671] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 1 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } }
I20260317 16:18:35.081509 21390 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:35.083071 21345 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:35.085707 21345 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:35.088353 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:35.089412 20238 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.091934 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:35.092880 20264 raft_consensus.cc:3060] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:35.093433 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:35.094066 20191 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 2
W20260317 16:18:35.100039 20163 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.102568 20015 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:35.107329 21164 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:35.072278 21277 raft_consensus.cc:740] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:35.107985 21164 raft_consensus.cc:697] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:35.108611 21277 consensus_queue.cc:260] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.109364 21164 consensus_queue.cc:237] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.110599 21277 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:35.114632 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.118147 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:35.120332 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:35.123471 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.124565 20268 raft_consensus.cc:2368] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
@ 0x7fea358db950 google::LogMessage::Flush() at ??:0
I20260317 16:18:35.126690 20272 raft_consensus.cc:1240] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
W20260317 16:18:35.119611 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 61845.4us
I20260317 16:18:35.129675 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:35.133126 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 272) took 1178 ms. Trace:
I20260317 16:18:35.133993 20265 raft_consensus.cc:2393] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:35.131340 21325 raft_consensus.cc:2749] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:35.131891 21216 consensus_queue.cc:1059] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:35.135504 21037 raft_consensus.cc:3055] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:35.135974 21037 raft_consensus.cc:740] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:35.137048 21037 consensus_queue.cc:260] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.138924 21037 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:35.132400 20016 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:35.133005 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.165045 20016 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:35.166263 21216 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:35.167035 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.167392 21325 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:35.133699 20041 rpcz_store.cc:276] 0317 16:18:33.954640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.451117 (+496477us) service_pool.cc:225] Handling call
0317 16:18:35.133034 (+681917us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:35.174933 21262 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:35.175644 21262 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:35.178720 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 86503.8us
I20260317 16:18:35.081835 19946 catalog_manager.cc:5671] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:35.198494 20191 raft_consensus.cc:2468] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:35.105412 21224 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:35.202564 20124 raft_consensus.cc:2468] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:35.241060 20238 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:35.241201 19946 catalog_manager.cc:5671] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 3, leader changed from 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195) to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 3 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } }
I20260317 16:18:35.244302 20238 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:35.245891 21345 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:35.246315 21345 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:35.246701 21345 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:35.182934 21262 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:35.179119 20269 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:35.246824 20321 tablet_bootstrap.cc:492] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:35.243279 19946 catalog_manager.cc:5671] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } }
I20260317 16:18:35.260093 20321 ts_tablet_manager.cc:1403] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.969s user 0.020s sys 0.022s
I20260317 16:18:35.256646 20238 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:35.263406 20042 raft_consensus.cc:2368] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
I20260317 16:18:35.263989 20043 raft_consensus.cc:1275] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260317 16:18:35.264583 19946 catalog_manager.cc:5671] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 2, leader changed from <none> to d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196). New cstate: current_term: 2 leader_uuid: "d8ed4b12dc654776aa7e64c0fa224321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } }
I20260317 16:18:35.266140 19946 catalog_manager.cc:5671] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:35.278486 21369 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.279294 21369 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:35.280864 21224 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:35.284797 21369 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:35.286542 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:35.288285 20191 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:35.240937 21006 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:35.294685 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.141299 20123 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:35.258942 21222 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:35.261906 21242 consensus_queue.cc:1048] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20260317 16:18:35.328097 21363 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:35.332490 21362 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:35.334503 21154 raft_consensus.cc:2749] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:35.335423 20264 raft_consensus.cc:2468] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:35.348706 19946 catalog_manager.cc:5671] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 4, leader changed from d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196) to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 4 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:35.329387 21369 raft_consensus.cc:2804] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:35.387256 21362 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:35.389753 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.390776 20193 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:35.343604 20116 raft_consensus.cc:1275] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260317 16:18:35.393028 20321 raft_consensus.cc:359] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:35.394033 20321 raft_consensus.cc:385] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:35.394380 20321 raft_consensus.cc:740] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:35.394938 20119 raft_consensus.cc:2468] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:35.400377 20320 tablet_bootstrap.cc:492] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:35.401093 20320 ts_tablet_manager.cc:1403] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.998s user 0.028s sys 0.054s
I20260317 16:18:35.402877 20041 raft_consensus.cc:2393] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 4: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:35.404271 20088 leader_election.cc:336] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.576s (SENT)
W20260317 16:18:35.405401 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 60722.8us
W20260317 16:18:35.406222 20088 leader_election.cc:336] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.605s (SENT)
I20260317 16:18:35.407155 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.408051 20043 raft_consensus.cc:2468] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:35.420401 21363 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:35.424485 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.425482 20044 raft_consensus.cc:2393] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 4: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:35.425611 20050 raft_consensus.cc:1240] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260317 16:18:35.426662 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.426914 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.427418 20048 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:35.429077 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.430085 20264 raft_consensus.cc:2468] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:35.406693 20088 leader_election.cc:304] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:35.453795 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:35.458287 20321 consensus_queue.cc:260] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:35.467474 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 493523us
I20260317 16:18:35.383383 19946 catalog_manager.cc:5671] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 1 to 3, leader changed from 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194) to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 3 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260317 16:18:35.489306 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:35.494602 20195 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.021s sys 0.044s
W20260317 16:18:35.463068 21115 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.023s sys 0.040s
I20260317 16:18:35.464834 21006 raft_consensus.cc:2749] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 16:18:35.496407 20195 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.022s sys 0.044s
I20260317 16:18:35.457608 21369 raft_consensus.cc:697] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:35.499938 20162 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.038s sys 0.030s
W20260317 16:18:35.504037 20162 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.038s sys 0.030s
I20260317 16:18:35.506136 20191 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
W20260317 16:18:35.511317 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:35.492367 20117 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.029s sys 0.072s
W20260317 16:18:35.497529 21115 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.023s sys 0.040s
I20260317 16:18:35.517386 20015 leader_election.cc:304] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:35.517984 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.485138 19946 catalog_manager.cc:5671] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 2, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 2 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: HEALTHY } } }
I20260317 16:18:35.528856 20320 raft_consensus.cc:359] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:35.529994 20320 raft_consensus.cc:385] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:35.530225 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.504413 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.530335 20320 raft_consensus.cc:740] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
W20260317 16:18:35.514637 20117 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.538s user 0.029s sys 0.073s
I20260317 16:18:35.535658 20321 ts_tablet_manager.cc:1434] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.242s user 0.008s sys 0.001s
W20260317 16:18:35.536587 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 130424us
W20260317 16:18:35.539242 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 548837us
I20260317 16:18:35.528872 21362 raft_consensus.cc:2804] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:35.537396 20320 consensus_queue.cc:260] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:35.540077 21157 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.050s sys 0.111s
W20260317 16:18:35.550560 21157 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.050s sys 0.111s
W20260317 16:18:35.552007 20088 leader_election.cc:336] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.651s (SENT)
I20260317 16:18:35.552644 20088 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:35.552961 20163 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.553771 21410 raft_consensus.cc:2749] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 16:18:35.543354 20125 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.008s sys 0.011s
W20260317 16:18:35.555244 20125 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.546s user 0.008s sys 0.012s
I20260317 16:18:35.558583 20321 tablet_bootstrap.cc:492] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:35.542851 20318 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.007s sys 0.013s
I20260317 16:18:35.539088 20043 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
W20260317 16:18:35.565315 21118 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.002s sys 0.026s
I20260317 16:18:35.564363 19946 catalog_manager.cc:5671] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195). New cstate: current_term: 1 leader_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } health_report { overall_health: HEALTHY } } }
W20260317 16:18:35.566596 20318 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.542s user 0.007s sys 0.013s
I20260317 16:18:35.565490 20193 raft_consensus.cc:1240] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:35.575305 20191 raft_consensus.cc:1240] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:35.577021 20264 raft_consensus.cc:1275] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
W20260317 16:18:35.584337 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.585537 20320 ts_tablet_manager.cc:1434] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.184s user 0.010s sys 0.001s
I20260317 16:18:35.561206 21362 raft_consensus.cc:697] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:35.572149 21118 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.002s sys 0.027s
W20260317 16:18:35.613811 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 53662.6us
I20260317 16:18:35.616348 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:35.588917 21224 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.572286 21345 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.612650 21037 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:35.636232 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 307574us
I20260317 16:18:35.628435 20320 tablet_bootstrap.cc:492] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: Bootstrap starting.
W20260317 16:18:35.629292 20162 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:35.640425 20238 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.634192 20195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 195) took 1097 ms. Trace:
I20260317 16:18:35.612046 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.642838 20238 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:35.636427 20041 raft_consensus.cc:2393] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:35.642522 20195 rpcz_store.cc:276] 0317 16:18:34.536558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:34.783812 (+247254us) service_pool.cc:225] Handling call
0317 16:18:34.792775 (+ 8963us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:34.912372 (+119597us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:34.912415 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:34.939577 (+ 27162us) log.cc:844] Serialized 26 byte log entry
0317 16:18:35.499735 (+560158us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:35.499787 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:35.554205 (+ 54418us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54001c75d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:35.554276 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:35.633765 (+ 79489us) raft_consensus.cc:1685] finished
0317 16:18:35.633797 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:35.634103 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3902,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6953,"spinlock_wait_cycles":72178432,"thread_start_us":559477,"threads_started":1,"wal-append.queue_time_us":691785}
I20260317 16:18:35.656687 21262 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:35.659708 21262 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:35.603950 21369 consensus_queue.cc:237] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:35.679759 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 68862.7us
W20260317 16:18:35.664351 21342 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.004s sys 0.025s
I20260317 16:18:35.637646 21224 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.644145 21037 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.670143 21262 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:35.702275 20091 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:35.686571 21216 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.012s sys 0.034s
W20260317 16:18:35.689553 21342 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s user 0.005s sys 0.025s
I20260317 16:18:35.677193 21222 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:35.689765 20190 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.002s sys 0.015s
I20260317 16:18:35.712946 20165 leader_election.cc:400] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
I20260317 16:18:35.719121 20165 leader_election.cc:403] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
W20260317 16:18:35.718994 20190 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.002s sys 0.015s
I20260317 16:18:35.717546 21222 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:35.729015 20091 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:35.711575 21216 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s user 0.012s sys 0.035s
I20260317 16:18:35.671284 21362 consensus_queue.cc:237] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:35.738452 20165 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:35.742518 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 169139us
W20260317 16:18:35.747076 20236 leader_election.cc:336] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.571s (SENT)
W20260317 16:18:35.754881 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 91940.7us
I20260317 16:18:35.703282 21409 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.081s
I20260317 16:18:35.767155 20165 leader_election.cc:304] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:35.743346 21325 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.026s sys 0.043s
I20260317 16:18:35.749480 21216 raft_consensus.cc:697] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:35.780570 21216 consensus_queue.cc:237] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.783015 21235 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:35.727871 21412 consensus_queue.cc:1059] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260317 16:18:35.788050 21416 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:35.806617 20321 tablet_bootstrap.cc:654] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:35.800268 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 74250.1us
W20260317 16:18:35.793475 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 121560us
I20260317 16:18:35.822587 20320 tablet_bootstrap.cc:654] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:35.788647 21427 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.837131 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:35.856139 20123 raft_consensus.cc:2393] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:35.856356 21412 raft_consensus.cc:3055] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:35.857582 21412 raft_consensus.cc:740] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:35.837841 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:35.839206 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:35.850070 20165 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:35.796746 21325 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.026s sys 0.044s
I20260317 16:18:35.787909 21390 raft_consensus.cc:2804] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:35.790848 21333 consensus_queue.cc:1059] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260317 16:18:35.852512 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:35.825841 21345 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:35.852699 21410 consensus_queue.cc:1059] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:35.788172 21407 raft_consensus.cc:2749] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213) with higher term. Message: Invalid argument: T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2.
I20260317 16:18:35.853375 21235 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.853917 21037 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:35.854341 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 120169us
I20260317 16:18:35.868942 20124 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:35.870102 21277 raft_consensus.cc:493] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.869699 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.869876 21325 raft_consensus.cc:697] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
W20260317 16:18:35.869701 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 48875.6us
W20260317 16:18:35.875480 20162 leader_election.cc:336] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.888s (SENT)
I20260317 16:18:35.904361 20162 leader_election.cc:304] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:35.877534 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:35.883716 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 92660.2us
W20260317 16:18:35.891794 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 162855us
I20260317 16:18:35.871101 20318 tablet_bootstrap.cc:492] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f: No bootstrap required, opened a new log
I20260317 16:18:35.906893 20318 ts_tablet_manager.cc:1403] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f: Time spent bootstrapping tablet: real 1.105s user 0.022s sys 0.013s
I20260317 16:18:35.892125 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.874449 21157 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:35.882184 20196 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:35.911679 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.877655 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:35.916771 21037 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:35.917423 21037 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:35.922094 21037 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:35.923254 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:35.924602 20091 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:35.929834 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 52395.5us
I20260317 16:18:35.874179 20121 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:35.884871 21390 raft_consensus.cc:697] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:35.868213 21427 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:35.935945 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:35.937337 20049 raft_consensus.cc:2468] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:35.938925 20272 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
I20260317 16:18:35.908576 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:35.940661 20088 leader_election.cc:336] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.729s (SENT)
I20260317 16:18:35.942134 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.947726 20091 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:35.911847 21224 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:35.950134 20165 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:35.903734 21277 raft_consensus.cc:515] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:35.892150 21235 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:35.941691 21407 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:35.937160 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:35.924615 21409 consensus_queue.cc:1059] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260317 16:18:35.943231 21408 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:35.967453 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:35.968506 20239 leader_election.cc:336] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Timed out: RequestConsensusVote RPC to 127.16.123.194:35679 timed out after 1.789s (SENT)
I20260317 16:18:35.972833 20239 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:35.955557 21312 raft_consensus.cc:2749] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:35.949260 21333 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
@ 0x7fea358e0a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260317 16:18:35.950070 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:35.970809 20091 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:35.975879 21417 log.cc:927] Time spent T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.222s user 0.002s sys 0.000s
I20260317 16:18:35.976104 20045 raft_consensus.cc:1240] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:35.987633 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.988458 20199 raft_consensus.cc:2468] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:35.971148 21118 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:35.889575 21412 consensus_queue.cc:260] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:35.967379 20269 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:35.996016 20090 leader_election.cc:304] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:35.963207 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:35.998006 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 125466us
I20260317 16:18:35.991811 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:35.970340 21006 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election won for term 5
W20260317 16:18:35.983341 21420 log.cc:927] Time spent T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260317 16:18:36.012549 21437 raft_consensus.cc:2804] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:36.013077 21437 raft_consensus.cc:493] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:36.013303 21429 log.cc:927] Time spent T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20260317 16:18:36.013417 21437 raft_consensus.cc:3060] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:35.999794 20197 raft_consensus.cc:1240] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:36.002409 20198 raft_consensus.cc:1275] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:36.003019 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.022678 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.003716 20041 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:36.004442 20091 leader_election.cc:304] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:36.005985 21416 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:35.993614 21118 raft_consensus.cc:493] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:36.007617 21412 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:36.010550 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:36.011839 21369 raft_consensus.cc:2749] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:36.012562 20318 raft_consensus.cc:359] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.036994 20318 raft_consensus.cc:385] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:35.974800 21408 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.021167 21006 raft_consensus.cc:697] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:35.988087 21210 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260317 16:18:36.033018 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 99159.5us
W20260317 16:18:36.023450 21419 log.cc:927] Time spent T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260317 16:18:36.037307 20318 raft_consensus.cc:740] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Initialized, Role: FOLLOWER
I20260317 16:18:36.058590 20318 consensus_queue.cc:260] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.036230 21118 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:36.023701 20045 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
W20260317 16:18:36.074831 20088 leader_election.cc:336] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.750s (SENT)
I20260317 16:18:35.995867 21363 consensus_queue.cc:1048] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20260317 16:18:36.081261 20236 leader_election.cc:400] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319) with higher term. Message: Invalid argument: T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4.
I20260317 16:18:36.055371 21210 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.092935 20088 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.048651 21408 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:35.979183 21325 consensus_queue.cc:237] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.055160 21037 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.054287 21333 raft_consensus.cc:2804] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:36.114619 20239 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:36.054889 21416 consensus_queue.cc:1048] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260317 16:18:36.119122 20318 ts_tablet_manager.cc:1434] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f: Time spent starting tablet: real 0.211s user 0.008s sys 0.000s
I20260317 16:18:36.103667 21210 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:36.118191 20162 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:36.126571 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 75863.6us
I20260317 16:18:36.112167 21329 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:36.103077 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.109318 21333 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.058869 21390 consensus_queue.cc:237] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.137573 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.150090 21216 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:36.150831 21216 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.154510 21216 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:36.154749 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.157325 20016 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.140962 20195 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:36.160889 20090 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:36.130340 21410 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:36.169186 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:36.171339 20041 raft_consensus.cc:2376] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2. Re-sending same reply.
I20260317 16:18:36.169411 20236 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:36.141664 21333 raft_consensus.cc:3060] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:36.161901 21312 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:36.158601 21277 leader_election.cc:290] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:36.118218 21037 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.163116 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.136034 21222 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.182992 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.112396 21006 consensus_queue.cc:237] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:36.173715 21427 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:36.188652 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 85483.3us
I20260317 16:18:36.189491 20162 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.169100 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:36.192953 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260317 16:18:36.217187 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 59974.2us
W20260317 16:18:36.218112 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 145246us
I20260317 16:18:36.183915 21345 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 16:18:36.221028 20162 leader_election.cc:343] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:36.172904 21404 consensus_queue.cc:1048] T 1234018d42ac449782301c404118e5bb P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.114s
I20260317 16:18:36.183615 21216 raft_consensus.cc:2749] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:36.180562 21235 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:36.225422 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 110904us
I20260317 16:18:36.225652 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.234800 20041 raft_consensus.cc:2393] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 6: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:36.186065 21410 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.234794 20088 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:36.235414 21410 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.238324 21410 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:36.231403 20321 tablet_bootstrap.cc:492] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:36.216722 21277 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:36.239601 20321 ts_tablet_manager.cc:1403] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.681s user 0.031s sys 0.015s
I20260317 16:18:36.239771 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.240444 20191 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:36.241542 20090 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:36.242391 20321 raft_consensus.cc:359] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:36.232774 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 69174us
I20260317 16:18:36.244911 20236 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:36.243844 20321 raft_consensus.cc:385] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:36.247206 20321 raft_consensus.cc:740] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:36.248404 20321 consensus_queue.cc:260] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.245045 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.251055 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.251837 20041 raft_consensus.cc:2468] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
I20260317 16:18:36.252951 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:36.253895 20041 raft_consensus.cc:2393] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:36.228729 20016 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.256779 20016 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:36.258993 21427 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:36.263165 21164 raft_consensus.cc:2749] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:36.262295 21277 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:36.265136 20016 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:36.271478 21164 raft_consensus.cc:2749] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:36.273510 20016 leader_election.cc:400] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Vote denied by peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679) with higher term. Message: Invalid argument: T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5.
I20260317 16:18:36.275276 20016 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:36.278733 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 136411us
I20260317 16:18:36.282060 21427 raft_consensus.cc:2749] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:36.285046 21422 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:36.285578 21422 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:36.285948 21422 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:36.192744 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:36.226948 20266 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6. Ops: []
W20260317 16:18:36.305526 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 58568.6us
I20260317 16:18:36.277233 20321 ts_tablet_manager.cc:1434] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s
I20260317 16:18:36.277796 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:36.307101 20162 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:36.309921 21262 consensus_meta.cc:287] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.594s user 0.010s sys 0.003s
I20260317 16:18:36.311580 21340 raft_consensus.cc:2749] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260317 16:18:36.312798 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 71333.4us
W20260317 16:18:36.313894 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 82936.6us
I20260317 16:18:36.314208 20239 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:36.309095 20041 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 6: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:36.227011 21450 raft_consensus.cc:2804] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:36.182621 21438 consensus_queue.cc:1059] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.233s
I20260317 16:18:36.300611 21235 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:36.324203 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 79468.9us
I20260317 16:18:36.336951 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.338128 20041 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:36.310381 21262 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:36.347545 20236 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:36.313089 21363 raft_consensus.cc:2749] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:36.313681 21164 raft_consensus.cc:2749] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:36.323644 21452 raft_consensus.cc:493] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:36.352411 20088 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:36.353003 21434 raft_consensus.cc:3055] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:36.353124 21424 consensus_queue.cc:1048] T 4c55cdb999654153a4cb90e34cb77217 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.178s
W20260317 16:18:36.354729 20150 heartbeater.cc:646] Failed to heartbeat to 127.16.123.254:35301 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.123.254:35301 timed out after 15.000s (SENT)
W20260317 16:18:36.355986 21447 log.cc:927] Time spent T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20260317 16:18:36.320436 21450 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.345431 21438 consensus_queue.cc:1243] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer 9a3afb098d6a45f0b405566771a1157b log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260317 16:18:36.456609 20239 leader_election.cc:304] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:36.351302 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 76073.8us
I20260317 16:18:36.389431 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.324822 21416 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:36.324359 21410 raft_consensus.cc:2804] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:36.320642 20045 raft_consensus.cc:2468] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:36.458250 20041 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:36.442341 21340 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.443368 21434 raft_consensus.cc:740] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:36.460362 21444 log.cc:927] Time spent T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.353s user 0.002s sys 0.000s
I20260317 16:18:36.477327 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.480919 20041 raft_consensus.cc:2376] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:36.482019 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:36.482626 20041 raft_consensus.cc:2376] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3. Re-sending same reply.
I20260317 16:18:36.483592 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.484145 20041 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:36.456481 21427 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.459710 21416 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.349395 21235 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.459880 21450 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:36.502262 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.467133 21037 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:36.465169 21410 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:36.506166 20195 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
W20260317 16:18:36.507953 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 155643us
W20260317 16:18:36.509323 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 173560us
I20260317 16:18:36.514933 20015 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:36.516350 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:36.452078 21452 raft_consensus.cc:515] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.519788 20091 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.490795 21363 raft_consensus.cc:3055] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:36.487815 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:36.505096 21416 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:36.480386 21340 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.510672 21224 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.516983 21410 raft_consensus.cc:3060] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:36.537655 20236 leader_election.cc:343] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:36.552323 20236 leader_election.cc:304] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:36.542941 21462 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.543687 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:36.545207 21453 raft_consensus.cc:493] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.545779 20271 raft_consensus.cc:1275] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:36.559242 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:36.559516 20320 tablet_bootstrap.cc:492] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: No bootstrap required, opened a new log
I20260317 16:18:36.560005 20320 ts_tablet_manager.cc:1403] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent bootstrapping tablet: real 0.932s user 0.032s sys 0.016s
I20260317 16:18:36.560595 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:36.561404 20271 raft_consensus.cc:2393] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:36.562178 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:36.540164 20118 raft_consensus.cc:1275] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:36.563827 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:36.553956 21438 consensus_queue.cc:1059] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260317 16:18:36.564827 20266 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:36.565178 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:36.566339 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:36.566444 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:36.567163 21455 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:36.545954 21432 raft_consensus.cc:2804] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:36.567440 20320 raft_consensus.cc:359] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.567916 21432 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:36.568243 20320 raft_consensus.cc:385] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:36.568326 21432 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:36.568555 20320 raft_consensus.cc:740] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Initialized, Role: FOLLOWER
I20260317 16:18:36.569219 20320 consensus_queue.cc:260] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.546293 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:36.546604 21441 log.cc:927] Time spent T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.276s user 0.000s sys 0.001s
I20260317 16:18:36.547371 20270 raft_consensus.cc:1275] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:36.504918 21427 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.571345 20320 ts_tablet_manager.cc:1434] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260317 16:18:36.547041 21451 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:36.572347 21451 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.572813 21451 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:36.573359 21427 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:36.529431 21443 log.cc:927] Time spent T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.356s user 0.001s sys 0.001s
I20260317 16:18:36.523388 21222 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:36.525147 21363 raft_consensus.cc:740] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:36.563599 21462 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:36.525792 21369 raft_consensus.cc:2804] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:36.529909 21006 raft_consensus.cc:2749] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:36.604296 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.605011 20191 raft_consensus.cc:2468] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
W20260317 16:18:36.603579 20196 consensus_meta.cc:287] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.663s user 0.012s sys 0.000s
I20260317 16:18:36.606600 20196 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:36.606838 20015 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:36.607928 20238 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:36.607275 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 305153us
W20260317 16:18:36.608745 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 89226.7us
I20260317 16:18:36.568571 21262 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:36.620908 21406 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.621726 20044 raft_consensus.cc:3060] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:36.537788 21434 consensus_queue.cc:260] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:36.622654 21434 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:36.623550 20049 raft_consensus.cc:1275] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:36.608884 21164 raft_consensus.cc:2804] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:36.621578 21406 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.555014 21224 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:36.633983 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 93868.5us
I20260317 16:18:36.646261 21280 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:36.601383 21369 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.647154 21369 raft_consensus.cc:3060] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:36.646976 21280 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.648747 20016 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:36.663928 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 312865us
I20260317 16:18:36.613306 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:36.627581 21164 raft_consensus.cc:493] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.627249 21277 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.633929 21404 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.651741 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:36.690560 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:36.692028 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:36.583491 21453 raft_consensus.cc:515] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.701720 20016 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:36.679492 21448 log.cc:927] Time spent T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.351s user 0.002s sys 0.000s
I20260317 16:18:36.690660 21423 raft_consensus.cc:2804] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:36.697813 21340 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:36.699644 21280 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:36.703308 20016 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.679333 21164 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:36.707747 21412 consensus_meta.cc:287] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.663s user 0.006s sys 0.007s
I20260317 16:18:36.717288 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.696125 21404 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.717978 20191 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
W20260317 16:18:36.705235 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 167611us
I20260317 16:18:36.719825 20015 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:36.686721 21262 raft_consensus.cc:2749] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:36.683925 21424 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election won for term 5
W20260317 16:18:36.696403 21118 consensus_meta.cc:287] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.613s user 0.012s sys 0.000s
I20260317 16:18:36.711884 21235 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:36.712332 21224 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:36.712468 21210 consensus_meta.cc:287] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.592s user 0.011s sys 0.001s
I20260317 16:18:36.704579 20192 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:36.707525 21423 raft_consensus.cc:493] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.686658 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:36.680420 21363 consensus_queue.cc:260] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.773084 21363 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:36.773536 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.723425 20236 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:36.719358 21404 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:36.743022 21424 raft_consensus.cc:697] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:36.743884 21477 log.cc:927] Time spent T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260317 16:18:36.743115 21472 raft_consensus.cc:2749] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:36.737378 21118 raft_consensus.cc:515] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.745009 21235 consensus_queue.cc:1048] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:36.689662 21277 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.756393 21423 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:36.749671 21224 raft_consensus.cc:493] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.757858 21474 consensus_queue.cc:1059] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:36.749614 21455 raft_consensus.cc:697] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:36.821379 21118 leader_election.cc:290] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:36.820581 21455 consensus_queue.cc:237] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.823114 20193 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: []
W20260317 16:18:36.762444 21437 consensus_meta.cc:287] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.748s user 0.015s sys 0.000s
I20260317 16:18:36.826458 21437 raft_consensus.cc:515] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:36.766192 21312 consensus_meta.cc:287] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.584s user 0.004s sys 0.008s
I20260317 16:18:36.769972 21434 consensus_queue.cc:1059] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:36.774895 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:36.832685 20050 raft_consensus.cc:2376] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2. Re-sending same reply.
I20260317 16:18:36.775660 21211 raft_consensus.cc:3055] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:36.777153 20122 raft_consensus.cc:1275] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:36.687268 21333 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.708320 21037 consensus_queue.cc:1048] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
W20260317 16:18:36.781414 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:36.781675 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 233870us
W20260317 16:18:36.790128 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:36.814195 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 187288us
I20260317 16:18:36.780951 20194 raft_consensus.cc:1275] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:36.754216 21210 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.763010 21457 consensus_queue.cc:1048] T ead98604210744a2b26b15fb54b5ea29 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260317 16:18:36.829445 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:36.849534 20125 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:36.850585 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.839823 21312 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:36.830698 21452 leader_election.cc:290] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:36.839025 21211 raft_consensus.cc:740] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:36.829054 21224 raft_consensus.cc:515] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.854755 20116 raft_consensus.cc:2393] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:36.847316 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.830641 21164 raft_consensus.cc:515] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.868916 21410 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:36.869879 21467 raft_consensus.cc:2804] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:36.870121 21480 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:36.870287 21210 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:36.868947 20199 raft_consensus.cc:1240] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: []
W20260317 16:18:36.868345 21483 consensus_queue.cc:1175] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "795c49e9595f45bdb8d6b568498abf1f" responder_term: 5 status { last_received { term: 5 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 5 index: 3 } }
I20260317 16:18:36.878229 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:36.884128 21479 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:36.884928 21479 raft_consensus.cc:2804] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:36.885296 21479 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:36.885612 21479 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:36.842549 21369 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:36.905905 20119 raft_consensus.cc:1240] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:36.906584 21333 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:36.891896 21408 consensus_meta.cc:287] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.778s user 0.009s sys 0.005s
I20260317 16:18:36.876559 21462 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
@ 0x7fea3f85ce4e kudu::TestWorkload::Setup() at ??:0
@ 0x354cc8 kudu::master::LeaderRebalancerTest::CreateWorkloadTable() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:110
@ 0x3515b6 kudu::master::LeaderRebalancerTest_RestartTserver_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:?
I20260317 16:18:36.881397 21467 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:36.967008 21467 raft_consensus.cc:3060] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:36.968971 21437 leader_election.cc:290] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:36.969909 21437 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:36.917800 21333 consensus_queue.cc:1048] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:36.890717 21235 raft_consensus.cc:3055] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Stepping down as leader of term 5
W20260317 16:18:36.927883 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 261860us
I20260317 16:18:36.935961 21455 consensus_queue.cc:1048] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:36.936472 21210 raft_consensus.cc:493] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:36.932291 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 228901us
I20260317 16:18:36.993453 20016 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.994702 20016 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:36.924750 21408 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.955971 21224 leader_election.cc:290] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:36.970413 21437 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.959108 21164 leader_election.cc:290] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:36.970979 21211 consensus_queue.cc:260] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.018181 21211 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:36.997996 21333 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:37.020280 21333 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.021229 21410 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:37.022040 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:37.022693 20118 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:37.022823 21329 consensus_meta.cc:287] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.887s user 0.013s sys 0.001s
I20260317 16:18:37.024613 21408 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:36.963503 21481 log.cc:927] Time spent T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260317 16:18:36.997936 20194 raft_consensus.cc:1240] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:36.998294 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:37.046547 20269 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:36.998477 21037 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:36.948839 20192 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:37.001333 20016 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:36.987432 21312 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:37.048163 21312 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:36.957520 20045 raft_consensus.cc:1275] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:37.048699 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:36.897673 21424 consensus_queue.cc:237] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:37.014379 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:37.014551 21224 raft_consensus.cc:2804] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:37.060300 21224 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:37.060803 21224 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:37.060380 20270 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:37.006143 21210 raft_consensus.cc:515] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:37.063318 21422 consensus_meta.cc:287] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.777s user 0.014s sys 0.002s
I20260317 16:18:36.980851 21235 raft_consensus.cc:740] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:37.015710 21325 consensus_queue.cc:1048] T a9f666d8f908427cb4523d57a45eb2b3 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:37.016242 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.078449 20122 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:37.080062 21312 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:36.991742 21216 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:37.031807 20195 raft_consensus.cc:2393] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:36.981331 21453 leader_election.cc:290] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:37.048657 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:36.895468 21480 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:37.081291 20194 raft_consensus.cc:3060] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:37.025732 21472 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:37.079991 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:36.951568 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 231391us
I20260317 16:18:37.034870 20049 raft_consensus.cc:1240] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260317 16:18:37.031172 21277 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:37.070008 21235 consensus_queue.cc:260] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:37.049552 20192 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
I20260317 16:18:37.087137 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
W20260317 16:18:37.089231 21451 consensus_meta.cc:287] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.516s user 0.009s sys 0.000s
I20260317 16:18:37.160331 20268 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 6: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:37.160156 21451 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:37.161104 20268 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 200) took 1235 ms (client timeout 1608 ms). Trace:
W20260317 16:18:37.161504 20268 rpcz_store.cc:269] 0317 16:18:35.925835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.935713 (+ 9878us) service_pool.cc:225] Handling call
0317 16:18:37.087013 (+1151300us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:37.161005 (+ 73992us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13136768}
I20260317 16:18:37.020299 21450 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.113265 21333 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:37.115114 21472 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.064383 21210 leader_election.cc:290] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:37.026527 21416 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:37.165802 20121 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:37.167068 21450 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:37.168244 21416 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:37.046571 21484 consensus_queue.cc:1048] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.203s
I20260317 16:18:37.169798 21369 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:37.170415 20044 consensus_meta.cc:287] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.548s user 0.010s sys 0.002s
I20260317 16:18:37.064391 21422 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.173034 20271 raft_consensus.cc:1240] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 4. Current term is 5. Ops: []
W20260317 16:18:37.173945 21432 consensus_meta.cc:287] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.605s user 0.010s sys 0.005s
I20260317 16:18:37.175726 21422 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:37.163115 20122 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:37.178972 21406 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:37.184994 21480 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:37.186785 21235 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:37.199016 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 183752us
I20260317 16:18:37.201154 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 234) took 1056 ms. Trace:
W20260317 16:18:37.203557 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 202277us
I20260317 16:18:37.204432 20016 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:37.207377 21409 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:37.208483 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:37.212293 20162 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:37.201642 20049 rpcz_store.cc:276] 0317 16:18:36.144319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.999570 (+855251us) service_pool.cc:225] Handling call
0317 16:18:37.033180 (+ 33610us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:37.085747 (+ 52567us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.201098 (+115351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:37.210119 20125 raft_consensus.cc:2410] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260317 16:18:37.225850 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:37.162514 21451 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:37.177740 21467 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:37.249473 20090 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:37.305904 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 56558.4us
I20260317 16:18:37.198490 20194 raft_consensus.cc:2468] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:37.301554 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:37.303779 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:37.303997 21164 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:37.304874 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 92697.7us
I20260317 16:18:37.229054 20270 raft_consensus.cc:1275] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:37.317173 20165 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:37.308097 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:37.219535 21325 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:37.314533 20162 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:37.307011 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.325500 20090 leader_election.cc:304] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:37.328320 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 278) took 1391 ms. Trace:
I20260317 16:18:37.216432 21409 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.328764 20048 rpcz_store.cc:276] 0317 16:18:35.933154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.939654 (+ 6500us) service_pool.cc:225] Handling call
0317 16:18:37.301355 (+1361701us) spinlock_profiling.cc:243] Waited 7.65 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:37.324295 (+ 22940us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17586048}
I20260317 16:18:37.174549 21432 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.304530 21453 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:37.315668 21340 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:37.324223 21406 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:37.318617 21480 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:37.319077 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:37.349318 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:37.340162 20162 leader_election.cc:343] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:37.318431 20267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 240) took 1189 ms. Trace:
I20260317 16:18:37.328579 21458 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:37.351230 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:37.324329 20269 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:37.319496 21164 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:37.345073 21406 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:37.354700 20272 raft_consensus.cc:1240] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:37.330873 21416 raft_consensus.cc:2804] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:37.335366 21277 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:37.326107 21210 consensus_queue.cc:1059] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260317 16:18:37.343863 21437 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:37.367579 21492 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:37.344642 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:37.375427 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:37.355890 21458 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:37.378989 20194 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:37.381309 21409 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:37.382992 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:37.418339 20270 raft_consensus.cc:2393] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:37.384106 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:37.343314 21453 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:37.387517 21493 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:37.392279 20268 raft_consensus.cc:2468] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:37.364885 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:37.397434 21164 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:37.424831 21423 consensus_meta.cc:287] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.597s user 0.006s sys 0.006s
I20260317 16:18:37.399658 21404 raft_consensus.cc:3055] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
W20260317 16:18:37.401023 21497 log.cc:927] Time spent T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260317 16:18:37.401510 20197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55464 (request call id 239) took 1403 ms. Trace:
I20260317 16:18:37.378499 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:37.388628 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:37.346127 21480 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:37.359247 20043 raft_consensus.cc:2468] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:37.375383 21416 raft_consensus.cc:697] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:37.448671 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.449440 20120 raft_consensus.cc:2468] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:37.449560 21416 consensus_queue.cc:237] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:37.451200 20165 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:37.384537 21468 raft_consensus.cc:3055] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:37.344246 21479 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:37.433346 20091 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.043s sys 0.067s
W20260317 16:18:37.459100 20091 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s user 0.044s sys 0.067s
W20260317 16:18:37.459401 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 611654us
I20260317 16:18:37.458575 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.461006 20124 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:37.461647 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 211) took 1097 ms. Trace:
I20260317 16:18:37.412683 21458 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:37.377275 20118 raft_consensus.cc:2468] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 6.
I20260317 16:18:37.398530 21406 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:37.437747 20125 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:37.436311 20197 rpcz_store.cc:276] 0317 16:18:35.986934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.987105 (+ 171us) service_pool.cc:225] Handling call
0317 16:18:35.987486 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:36.015281 (+ 27795us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.385196 (+1369915us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b54002125d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:37.390701 (+ 5505us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27481856}
W20260317 16:18:37.427012 21412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.005s sys 0.063s
I20260317 16:18:37.421222 21224 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:37.452831 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:37.462026 20124 rpcz_store.cc:276] 0317 16:18:36.364313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.364547 (+ 234us) service_pool.cc:225] Handling call
0317 16:18:37.458403 (+1093856us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:37.461573 (+ 3170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8694528}
I20260317 16:18:37.437896 21404 raft_consensus.cc:740] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:37.473042 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7faa93fffc94522a88d5c47721186bd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:37.539904 20197 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:37.474617 21450 consensus_queue.cc:1059] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20260317 16:18:37.451751 21480 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:37.582798 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.583539 20120 raft_consensus.cc:2393] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:37.448711 20043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 275) took 1534 ms (client timeout 1822 ms). Trace:
I20260317 16:18:37.433077 21493 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:37.470221 21154 raft_consensus.cc:2804] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260317 16:18:37.475203 21412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.005s sys 0.063s
I20260317 16:18:37.582157 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.580268 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 235) took 1500 ms. Trace:
I20260317 16:18:37.455189 21468 raft_consensus.cc:740] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
@ 0x7fea36442e30 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fea36420a32 testing::Test::Run() at ??:0
I20260317 16:18:37.439360 21423 raft_consensus.cc:515] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:37.613590 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
@ 0x7fea36421d49 testing::TestInfo::Run() at ??:0
W20260317 16:18:37.619009 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:37.534200 21340 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:37.479218 21467 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:37.640142 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:38.128170 20195 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:38.127691 21340 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.129559 21423 leader_election.cc:290] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.130267 21340 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.127894 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:37.588018 20165 leader_election.cc:304] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:38.131657 20124 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
W20260317 16:18:38.131696 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 544201us
I20260317 16:18:38.132761 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.133519 20124 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:38.134341 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:38.134517 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:37.537070 21491 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:37.655150 21406 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.141983 21406 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:37.664424 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.655550 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.146077 20119 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:37.624883 21210 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.141988 21491 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.147349 20120 raft_consensus.cc:1275] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:38.149744 21340 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.150059 21491 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:37.671057 20191 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.004s sys 0.022s
I20260317 16:18:38.150223 21340 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:38.150882 20191 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.236s user 0.005s sys 0.022s
I20260317 16:18:38.151706 20191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 338) took 1308 ms. Trace:
I20260317 16:18:38.152428 21340 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:37.666839 20236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.038s sys 0.076s
W20260317 16:18:38.153174 20236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.362s user 0.038s sys 0.076s
I20260317 16:18:38.152043 20191 rpcz_store.cc:276] 0317 16:18:36.843193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.843574 (+ 381us) service_pool.cc:225] Handling call
0317 16:18:36.853416 (+ 9842us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:36.853661 (+ 245us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:36.853698 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:36.865871 (+ 12173us) log.cc:844] Serialized 26 byte log entry
0317 16:18:38.151318 (+1285447us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:38.151365 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.151451 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:38.151516 (+ 65us) raft_consensus.cc:1685] finished
0317 16:18:38.151530 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.151651 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1236059,"threads_started":1,"wal-append.queue_time_us":913732}
W20260317 16:18:38.153759 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 1.43036e+06us
I20260317 16:18:38.154094 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:37.682636 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.154430 20236 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:38.155619 20236 leader_election.cc:343] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:38.156075 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.156605 20165 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:37.671380 20238 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.009s sys 0.027s
I20260317 16:18:37.703797 20194 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
W20260317 16:18:38.157230 20238 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.281s user 0.009s sys 0.027s
I20260317 16:18:37.604967 21154 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:38.157519 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 1.28183e+06us
I20260317 16:18:38.157819 21154 raft_consensus.cc:3060] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:38.157920 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:38.158906 20194 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:37.623992 21432 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:37.671741 20045 raft_consensus.cc:1240] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 3. Ops: []
I20260317 16:18:37.668884 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:37.714933 20088 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.011s sys 0.045s
W20260317 16:18:38.164357 20088 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.138s user 0.011s sys 0.045s
W20260317 16:18:38.164731 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 1.13829e+06us
W20260317 16:18:37.715350 20239 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.021s sys 0.047s
W20260317 16:18:37.738396 20050 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.016s sys 0.027s
W20260317 16:18:37.729455 20089 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.002s sys 0.035s
I20260317 16:18:38.164450 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 203) took 1520 ms. Trace:
W20260317 16:18:38.165549 20050 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.101s user 0.016s sys 0.028s
I20260317 16:18:37.705842 21479 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:37.755260 20041 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.010s sys 0.035s
I20260317 16:18:37.760995 20193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 206) took 1887 ms. Trace:
W20260317 16:18:38.168278 20041 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.340s user 0.010s sys 0.035s
I20260317 16:18:37.753207 20264 raft_consensus.cc:1240] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:37.690944 21404 consensus_queue.cc:260] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:37.773285 20122 consensus_meta.cc:287] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.596s user 0.012s sys 0.000s
I20260317 16:18:37.779791 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:37.782742 21216 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:37.751835 21453 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:37.722191 21468 consensus_queue.cc:260] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.174974 20122 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
W20260317 16:18:37.792407 21235 consensus_meta.cc:287] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.605s user 0.012s sys 0.001s
W20260317 16:18:37.745649 21474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.896s user 0.005s sys 0.043s
I20260317 16:18:38.177582 20122 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46060 (request call id 223) took 1177 ms. Trace:
W20260317 16:18:38.177666 20123 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 213) took 1812 ms (client timeout 1979 ms). Trace:
I20260317 16:18:37.612742 20265 rpcz_store.cc:276] 0317 16:18:36.080096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.099212 (+ 19116us) service_pool.cc:225] Handling call
0317 16:18:36.135243 (+ 36031us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:36.192511 (+ 57268us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:36.216053 (+ 23542us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:36.334172 (+118119us) log.cc:844] Serialized 26 byte log entry
0317 16:18:36.527387 (+193215us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:36.537402 (+ 10015us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:36.548111 (+ 10709us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:37.456698 (+908587us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b440048f1f0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea361287fc 00007fea36128c57 00007fea38ec73d5 00007fea38ec7b09 00007fea38ec7d0a 00007fea38ec7edd 00007fea38e9f738 00007fea38e9ab0a 00007fea38e99f56 00007fea38e99b78
0317 16:18:37.469020 (+ 12322us) raft_consensus.cc:1685] finished
0317 16:18:37.469047 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.580189 (+111142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36424,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":5239,"spinlock_wait_cycles":154553216,"thread_start_us":56922,"threads_started":1,"wal-append.queue_time_us":137383}
I20260317 16:18:37.809285 20197 raft_consensus.cc:2468] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:37.794734 21224 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:37.353511 20267 rpcz_store.cc:276] 0317 16:18:36.127205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.131847 (+ 4642us) service_pool.cc:225] Handling call
0317 16:18:36.355410 (+223563us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:36.355620 (+ 210us) raft_consensus.cc:1471] Updating replica for 2 ops
0317 16:18:36.459298 (+103678us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:36.463027 (+ 3729us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0317 16:18:36.529087 (+ 66060us) log.cc:844] Serialized 52 byte log entry
0317 16:18:36.811157 (+282070us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:36.811756 (+ 599us) log.cc:844] Serialized 14 byte log entry
0317 16:18:36.991922 (+180166us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.007552 (+ 15630us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:37.208326 (+200774us) raft_consensus.cc:1685] finished
0317 16:18:37.237281 (+ 28955us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.316257 (+ 78976us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83862,"prepare.queue_time_us":389,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"raft.queue_time_us":141777,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":14750,"spinlock_wait_cycles":7469568,"thread_start_us":452461,"threads_started":2,"wal-append.queue_time_us":468530}
W20260317 16:18:37.815451 20016 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.008s sys 0.018s
W20260317 16:18:38.181093 20016 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.009s sys 0.018s
I20260317 16:18:37.818048 21493 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:37.824990 21439 log.cc:927] Time spent T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 1.644s user 0.018s sys 0.012s
W20260317 16:18:37.832790 21362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.713s user 0.024s sys 0.012s
I20260317 16:18:38.183009 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:38.183851 21362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.064s user 0.024s sys 0.012s
I20260317 16:18:38.178787 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 236) took 2098 ms. Trace:
I20260317 16:18:38.183907 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.184859 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:37.851249 20049 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.505s user 0.009s sys 0.028s
I20260317 16:18:38.185529 21493 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.178717 20117 raft_consensus.cc:2410] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260317 16:18:37.833207 20014 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.022s sys 0.049s
I20260317 16:18:38.180778 21340 raft_consensus.cc:2804] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260317 16:18:38.186334 20014 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.978s user 0.022s sys 0.050s
I20260317 16:18:38.186131 21493 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.186610 21340 raft_consensus.cc:493] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:38.187069 21340 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:37.851450 21211 consensus_meta.cc:287] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Time spent flushing consensus metadata: real 0.833s user 0.011s sys 0.003s
W20260317 16:18:37.853223 20116 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46060 (request call id 214) took 1711 ms (client timeout 1696 ms). Trace:
W20260317 16:18:37.849002 20266 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.004s sys 0.025s
W20260317 16:18:38.187971 20266 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.004s sys 0.025s
I20260317 16:18:38.188308 21493 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.188505 21519 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.189036 21519 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.190032 20270 raft_consensus.cc:1240] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
W20260317 16:18:37.856027 21501 log.cc:927] Time spent T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.120s user 0.003s sys 0.000s
I20260317 16:18:38.177098 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.184587 20264 rpcz_store.cc:276] 0317 16:18:36.080572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.083831 (+ 3259us) service_pool.cc:225] Handling call
0317 16:18:37.750589 (+1666758us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:37.752987 (+ 2398us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:38.174964 (+421977us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.178681 (+ 3717us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11376512}
W20260317 16:18:38.179669 20123 rpcz_store.cc:269] 0317 16:18:36.365164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.416819 (+ 51655us) service_pool.cc:225] Handling call
0317 16:18:37.779602 (+1362783us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:38.177606 (+398004us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9527040}
I20260317 16:18:37.861917 20190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 213) took 1322 ms. Trace:
I20260317 16:18:38.192270 20190 rpcz_store.cc:276] 0317 16:18:36.539223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.540940 (+ 1717us) service_pool.cc:225] Handling call
0317 16:18:37.385416 (+844476us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.388512 (+ 3096us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:37.388550 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:37.388602 (+ 52us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:37.388640 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.388720 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.861592 (+472872us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b54002125d0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:37.861826 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2672,"raft.queue_time_us":3206,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"spinlock_wait_cycles":1884780416}
I20260317 16:18:38.193884 20123 raft_consensus.cc:1275] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260317 16:18:38.194290 20123 pending_rounds.cc:85] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Aborting all ops after (but not including) 0
I20260317 16:18:37.852159 21458 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:37.870249 21344 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.184875 20268 raft_consensus.cc:2468] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 0.
I20260317 16:18:38.195636 21344 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.175971 21406 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:38.196837 21458 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.197706 21344 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:37.888973 20044 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 229) took 1758 ms. Trace:
I20260317 16:18:37.878074 20321 tablet_bootstrap.cc:492] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:38.198328 20044 rpcz_store.cc:276] 0317 16:18:36.130611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.621007 (+490396us) service_pool.cc:225] Handling call
0317 16:18:36.621466 (+ 459us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:37.305085 (+683619us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:37.314839 (+ 9754us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:37.342427 (+ 27588us) log.cc:844] Serialized 26 byte log entry
0317 16:18:37.839248 (+496821us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:37.840111 (+ 863us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.888625 (+ 48514us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:37.888706 (+ 81us) raft_consensus.cc:1685] finished
0317 16:18:37.888722 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.888903 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63451,"raft.queue_time_us":135068,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":14546,"thread_start_us":490065,"threads_started":1,"wal-append.queue_time_us":534576}
I20260317 16:18:38.200387 20197 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:38.200657 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.201455 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:38.197566 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:37.833815 21494 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.015s sys 0.036s
I20260317 16:18:37.878559 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46046 (request call id 267) took 1396 ms. Trace:
W20260317 16:18:37.897591 21325 consensus_meta.cc:287] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.565s user 0.009s sys 0.001s
I20260317 16:18:37.878805 21407 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:37.903519 21523 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:37.929183 21480 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:37.939291 20196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 215) took 1245 ms. Trace:
I20260317 16:18:37.968434 20273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 238) took 1865 ms. Trace:
I20260317 16:18:37.980858 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:38.209208 21523 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.210080 20192 raft_consensus.cc:2468] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:38.210964 21480 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:38.209707 20273 rpcz_store.cc:276] 0317 16:18:36.103475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.103841 (+ 366us) service_pool.cc:225] Handling call
0317 16:18:37.660063 (+1556222us) spinlock_profiling.cc:243] Waited 4.67 ms on lock 0x7b5800501698. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:37.660328 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.660600 (+ 272us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:37.871009 (+210409us) log.cc:844] Serialized 14 byte log entry
0317 16:18:37.967978 (+ 96969us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:37.968045 (+ 67us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:37.968134 (+ 89us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.968227 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.968335 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":235586,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":87,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":2462,"spinlock_wait_cycles":10732928,"thread_start_us":304720,"threads_started":2,"wal-append.queue_time_us":95208}
I20260317 16:18:37.991807 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 208) took 1346 ms. Trace:
I20260317 16:18:38.211326 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:38.017505 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 83e3771d4d494b8caa5d647d8073666c (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet ca1e9c0371b4438e8c45c1a54013c9d7
I20260317 16:18:38.211982 20192 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:38.209046 21519 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.206899 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.038022 19946 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46702 (request call id 36) took 9185 ms. Trace:
I20260317 16:18:38.064081 21280 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.213562 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.213862 21280 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.214259 20044 raft_consensus.cc:2376] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:38.128278 20042 raft_consensus.cc:2393] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 6: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:37.612820 21333 raft_consensus.cc:3055] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:38.206312 20121 rpcz_store.cc:276] 0317 16:18:36.482157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.567503 (+ 85346us) service_pool.cc:225] Handling call
0317 16:18:37.865649 (+1298146us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54002e2054. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:37.876660 (+ 11011us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1137596160}
I20260317 16:18:38.211755 20048 rpcz_store.cc:276] 0317 16:18:36.645832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:37.334597 (+688765us) service_pool.cc:225] Handling call
0317 16:18:37.334913 (+ 316us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.337330 (+ 2417us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:37.840570 (+503240us) log.cc:844] Serialized 14 byte log entry
0317 16:18:37.991378 (+150808us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:37.991441 (+ 63us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:37.991486 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.991572 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.991733 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8906,"prepare.queue_time_us":40731,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":181,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":151,"spinlock_wait_cycles":2304,"thread_start_us":631044,"threads_started":2,"wal-append.queue_time_us":146054}
I20260317 16:18:38.134632 21412 raft_consensus.cc:2804] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Leader election won for term 6
I20260317 16:18:37.660326 20199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 211) took 1503 ms. Trace:
I20260317 16:18:38.217267 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.217571 21412 raft_consensus.cc:697] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:38.217509 20199 rpcz_store.cc:276] 0317 16:18:36.157138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.157288 (+ 150us) service_pool.cc:225] Handling call
0317 16:18:36.157581 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:36.883440 (+725859us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.660068 (+776628us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b5400459a54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:37.660257 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1628984704}
I20260317 16:18:38.211673 21523 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:38.219103 20199 raft_consensus.cc:1275] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:38.218660 21412 consensus_queue.cc:237] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.219475 21340 raft_consensus.cc:515] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.147631 21210 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.134924 20124 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:38.156011 20236 leader_election.cc:304] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:38.220348 21280 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:38.221113 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 65519.4us
W20260317 16:18:37.670747 21488 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.013s sys 0.016s
W20260317 16:18:38.222054 21488 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.351s user 0.013s sys 0.016s
W20260317 16:18:37.663383 21006 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.049s sys 0.074s
I20260317 16:18:38.222754 21488 raft_consensus.cc:2749] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 16:18:38.222941 21006 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.292s user 0.050s sys 0.074s
I20260317 16:18:38.166543 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 206) took 1521 ms. Trace:
W20260317 16:18:37.585427 20043 rpcz_store.cc:269] 0317 16:18:35.914560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.924791 (+ 10231us) service_pool.cc:225] Handling call
0317 16:18:37.316620 (+1391829us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:37.448529 (+131909us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12135168}
I20260317 16:18:38.224025 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.224506 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.224797 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.225078 20268 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:38.224740 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:38.165494 20239 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.125s user 0.021s sys 0.047s
W20260317 16:18:38.226660 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 1.1859e+06us
I20260317 16:18:38.227092 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.223372 20050 rpcz_store.cc:276] 0317 16:18:36.645224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:37.063889 (+418665us) service_pool.cc:225] Handling call
0317 16:18:37.064150 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.064345 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:37.064663 (+ 318us) log.cc:844] Serialized 14 byte log entry
0317 16:18:38.166200 (+1101537us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:38.166256 (+ 56us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:38.166297 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.166362 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.166490 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":576,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":1238,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3467,"thread_start_us":1101160,"threads_started":1,"wal-append.queue_time_us":764244}
I20260317 16:18:38.228164 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.228294 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.227015 21406 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.229234 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.229173 21406 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.226402 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:38.230145 20239 connection.cc:232] RPC call timeout handler was delayed by -1.0442442620001202s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2331s for a response
I20260317 16:18:38.230398 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.162405 21489 raft_consensus.cc:493] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.235293 21489 raft_consensus.cc:515] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.237340 21489 leader_election.cc:290] T b7f7f224bb064944a05175b00d835fce P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:38.172410 20088 leader_election.cc:336] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.727s (SENT)
I20260317 16:18:38.209633 20196 rpcz_store.cc:276] 0317 16:18:36.693613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.693758 (+ 145us) service_pool.cc:225] Handling call
0317 16:18:36.694040 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:36.694233 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 4
0317 16:18:36.723889 (+ 29656us) log.cc:844] Serialized 14 byte log entry
0317 16:18:37.765804 (+1041915us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b1800148f00. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea361c74ca 00007fea3f8595c9 00007fea360871c0 00007fea3624f474 00007fea3624f3e5 00007fea3cd025f7 00007fea3930a8d7 00007fea39319fef 00007fea39319f68
0317 16:18:37.766024 (+ 220us) log.cc:844] Serialized 14 byte log entry
0317 16:18:37.938645 (+172621us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:37.938861 (+ 216us) raft_consensus.cc:1645] Marking committed up to 4
0317 16:18:37.938941 (+ 80us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.939038 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.939214 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1030064,"prepare.queue_time_us":1097442,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":1023,"raft.queue_time_us":17720,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":20999,"spinlock_wait_cycles":4588416,"thread_start_us":172211,"threads_started":1,"wal-append.queue_time_us":201909}
I20260317 16:18:38.221657 21210 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.239312 21280 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.239830 21280 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.241721 21280 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:38.168052 20089 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.343s user 0.003s sys 0.035s
I20260317 16:18:38.176069 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 222) took 2058 ms. Trace:
W20260317 16:18:38.177983 21474 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.331s user 0.005s sys 0.044s
W20260317 16:18:38.243530 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.178045 20122 rpcz_store.cc:276] 0317 16:18:36.999758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:37.079656 (+ 79898us) service_pool.cc:225] Handling call
0317 16:18:38.177523 (+1097867us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:37.820791 21472 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.181411 20015 leader_election.cc:304] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:38.244458 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:38.244630 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 65061us
I20260317 16:18:38.245520 20122 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:38.174990 20195 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:38.179525 21216 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:38.246790 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:38.247815 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:38.248107 20124 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:38.249104 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:38.170609 21432 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.250590 21432 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.251426 20265 raft_consensus.cc:1275] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260317 16:18:38.242667 20041 rpcz_store.cc:276] 0317 16:18:36.117808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.504784 (+386976us) service_pool.cc:225] Handling call
0317 16:18:36.778746 (+273962us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:36.786326 (+ 7580us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:36.786378 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:36.817708 (+ 31330us) log.cc:844] Serialized 26 byte log entry
0317 16:18:38.168835 (+1351127us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:38.168892 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.168980 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:38.169038 (+ 58us) raft_consensus.cc:1685] finished
0317 16:18:38.169054 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.175797 (+ 6743us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b5400067050. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:38.175999 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113130,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":1118,"spinlock_wait_cycles":629268864,"thread_start_us":1340023,"threads_started":1,"wal-append.queue_time_us":1011164}
I20260317 16:18:38.252668 21432 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:38.253108 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.253612 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:38.187850 20116 rpcz_store.cc:269] 0317 16:18:36.141752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.166579 (+ 24827us) service_pool.cc:225] Handling call
0317 16:18:37.852845 (+1686266us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b5400112454. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:37.853122 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1213505792}
I20260317 16:18:38.197638 20047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 282) took 2240 ms. Trace:
I20260317 16:18:38.185056 21423 raft_consensus.cc:493] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:37.894096 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:37.868003 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 283) took 1902 ms. Trace:
I20260317 16:18:38.194509 20123 pending_rounds.cc:107] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260317 16:18:38.255239 20125 raft_consensus.cc:2393] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:38.255295 20238 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:38.202593 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:38.208452 21325 raft_consensus.cc:2749] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 16:18:38.202462 21494 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.040s user 0.015s sys 0.036s
I20260317 16:18:38.206704 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.256315 21325 raft_consensus.cc:2804] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:38.256795 20267 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:38.206547 21458 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:38.256882 21325 raft_consensus.cc:697] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:38.257434 21458 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.257886 21325 consensus_queue.cc:237] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.036046 20198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34608 (request call id 209) took 2010 ms. Trace:
I20260317 16:18:38.215138 21333 raft_consensus.cc:740] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:38.221803 20236 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.156754 21467 raft_consensus.cc:2804] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:38.202600 20191 raft_consensus.cc:2393] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:38.163664 20118 raft_consensus.cc:2376] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4. Re-sending same reply.
I20260317 16:18:38.209113 21407 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.165946 20045 rpcz_store.cc:276] 0317 16:18:36.644006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:37.049477 (+405471us) service_pool.cc:225] Handling call
0317 16:18:37.059960 (+ 10483us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:38.162208 (+1102248us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.164221 (+ 2013us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b54001da954. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:38.164394 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1270850432}
I20260317 16:18:38.231693 20270 raft_consensus.cc:1240] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:38.291829 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.292279 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.292897 21407 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.293648 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.212910 19946 rpcz_store.cc:276] 0317 16:18:28.894302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:28.903923 (+ 9621us) service_pool.cc:225] Handling call
0317 16:18:38.037960 (+9134037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 16:18:36.219314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 16:18:36.506574 (+287260us) write_op.cc:432] Acquiring schema lock in shared mode
0317 16:18:36.518399 (+ 11825us) write_op.cc:435] Acquired schema lock
0317 16:18:36.522209 (+ 3810us) tablet.cc:661] Decoding operations
0317 16:18:36.565299 (+ 43090us) write_op.cc:620] Acquiring the partition lock for write op
0317 16:18:36.565384 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0317 16:18:36.565413 (+ 29us) tablet.cc:684] Acquiring locks for 12 operations
0317 16:18:36.566562 (+ 1149us) tablet.cc:700] Row locks acquired
0317 16:18:36.566577 (+ 15us) write_op.cc:260] PREPARE: finished
0317 16:18:36.566736 (+ 159us) write_op.cc:270] Start()
0317 16:18:36.620800 (+ 54064us) write_op.cc:276] Timestamp: P: 1773764316566677 usec, L: 0
0317 16:18:36.620866 (+ 66us) op_driver.cc:348] REPLICATION: starting
0317 16:18:36.718978 (+ 98112us) log.cc:844] Serialized 4845 byte log entry
0317 16:18:37.891479 (+1172501us) op_driver.cc:464] REPLICATION: finished
0317 16:18:37.965402 (+ 73923us) write_op.cc:301] APPLY: starting
0317 16:18:37.965510 (+ 108us) tablet.cc:1366] starting BulkCheckPresence
0317 16:18:37.971647 (+ 6137us) tablet.cc:1369] finished BulkCheckPresence
0317 16:18:37.971869 (+ 222us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 16:18:37.974127 (+ 2258us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 16:18:37.974591 (+ 464us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0317 16:18:37.974704 (+ 113us) write_op.cc:312] APPLY: finished
0317 16:18:37.992868 (+ 18164us) log.cc:844] Serialized 88 byte log entry
0317 16:18:37.993027 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:37.993129 (+ 102us) write_op.cc:454] Released schema lock
0317 16:18:37.993175 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0317 16:18:37.993492 (+ 317us) write_op.cc:489] Releasing partition, row and schema locks
0317 16:18:37.993510 (+ 18us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":72439,"apply.run_cpu_time_us":4578,"apply.run_wall_time_us":29193,"num_ops":12,"prepare.queue_time_us":186724,"prepare.run_cpu_time_us":34390,"prepare.run_wall_time_us":739038,"raft.queue_time_us":728763,"raft.run_cpu_time_us":18770,"raft.run_wall_time_us":73133,"replication_time_us":1270385,"thread_start_us":339382,"threads_started":4,"wal-append.queue_time_us":330029}]]}
I20260317 16:18:38.233412 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.218358 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.295123 20273 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:38.238336 20088 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:38.253727 20041 raft_consensus.cc:2468] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:38.171377 20194 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:38.254662 20266 raft_consensus.cc:2393] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 4: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:38.194527 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:38.170420 20193 rpcz_store.cc:276] 0317 16:18:35.873652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.982642 (+108990us) service_pool.cc:225] Handling call
0317 16:18:35.983080 (+ 438us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:36.823559 (+840479us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.760647 (+937088us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54001c75d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:37.760914 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1931431808}
I20260317 16:18:38.248776 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.229492 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.259140 20264 raft_consensus.cc:1275] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:38.282263 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.280797 21333 consensus_queue.cc:260] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.315603 20269 raft_consensus.cc:3060] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:38.315686 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.316387 20270 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:38.280702 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 9a97b610497d410fb77dc8d4e5aca57e (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 91810c44e7184392a8d2694ecf9228d7
I20260317 16:18:38.317210 20192 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:38.290884 21467 raft_consensus.cc:493] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:38.319255 21467 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:38.259233 20197 raft_consensus.cc:2368] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:38.235793 21154 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.290738 21519 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:38.292449 20045 raft_consensus.cc:3060] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:38.255532 20123 raft_consensus.cc:2889] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:38.297483 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:38.301443 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.301553 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.323338 20048 raft_consensus.cc:2393] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:38.323894 20268 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:38.301537 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.282064 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.267197 21425 raft_consensus.cc:493] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.302811 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.285264 21452 raft_consensus.cc:3055] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:38.303689 20015 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:38.352130 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 52139.6us
I20260317 16:18:38.304258 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.307013 20199 raft_consensus.cc:1275] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer 795c49e9595f45bdb8d6b568498abf1f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:38.262033 20321 tablet_bootstrap.cc:654] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:38.354184 20041 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:38.355311 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:38.308164 20016 connection.cc:232] RPC call timeout handler was delayed by -1.0306151320000936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260317 16:18:38.310842 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 138469us
I20260317 16:18:38.312237 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:38.299621 21523 consensus_queue.cc:1059] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260317 16:18:38.303290 21224 consensus_queue.cc:1048] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260317 16:18:38.309402 21473 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:38.314675 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.314945 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.315820 21333 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:38.286533 21340 leader_election.cc:290] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.290755 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:38.250234 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.352280 21262 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election won for term 4
W20260317 16:18:38.336684 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.337627 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:38.327126 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 71918.3us
I20260317 16:18:38.399422 20048 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:38.361955 20265 raft_consensus.cc:3060] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:38.258944 20198 rpcz_store.cc:276] 0317 16:18:36.025391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.025688 (+ 297us) service_pool.cc:225] Handling call
0317 16:18:37.350449 (+1324761us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:37.350647 (+ 198us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.350872 (+ 225us) raft_consensus.cc:1534] Early marking committed up to index 2
0317 16:18:37.945890 (+595018us) log.cc:844] Serialized 14 byte log entry
0317 16:18:38.035523 (+ 89633us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:38.035601 (+ 78us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:38.035666 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.035756 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.035947 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":442968,"prepare.queue_time_us":448386,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":113,"raft.queue_time_us":291,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":2654080,"thread_start_us":240751,"threads_started":2,"wal-append.queue_time_us":89484}
I20260317 16:18:38.351691 21452 raft_consensus.cc:740] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:38.357206 20088 leader_election.cc:336] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 2.034s (SENT)
I20260317 16:18:38.362825 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:38.363730 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 142001us
I20260317 16:18:38.422024 20236 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:38.353911 21425 raft_consensus.cc:515] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.308034 21423 raft_consensus.cc:515] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.392902 20124 raft_consensus.cc:2410] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:38.255079 20046 rpcz_store.cc:276] 0317 16:18:35.965475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.966304 (+ 829us) service_pool.cc:225] Handling call
0317 16:18:37.047531 (+1081227us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:37.866596 (+819065us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea3621518c 00007fea36214d4b 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:37.866970 (+ 374us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.867290 (+ 320us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:37.867348 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:37.867553 (+ 205us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:37.867609 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.867706 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:37.867908 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26815104}
I20260317 16:18:38.350459 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.427960 20050 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
W20260317 16:18:38.429407 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 72229us
W20260317 16:18:38.430294 20088 leader_election.cc:336] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 2.179s (SENT)
I20260317 16:18:38.362002 21479 raft_consensus.cc:2804] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:38.402480 20239 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:38.435846 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 80647.3us
I20260317 16:18:38.436622 20165 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:38.437456 21340 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:38.405133 21325 raft_consensus.cc:493] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
W20260317 16:18:38.409449 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 4ff46c37d04d446981ad8728f73c6e1f (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet e5f06855b51745fbb08acb481a5ad9bb
I20260317 16:18:38.386843 21280 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.412290 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.337095 21216 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.338362 21519 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.401113 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.475502 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:38.398495 20042 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56386 (request call id 273) took 2474 ms (client timeout 1602 ms). Trace:
I20260317 16:18:38.437493 20165 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:38.359985 20015 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:38.439920 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.362839 21362 consensus_queue.cc:1059] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
I20260317 16:18:38.400853 20118 raft_consensus.cc:2468] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:38.415549 21262 raft_consensus.cc:697] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:38.480036 20043 raft_consensus.cc:3060] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
W20260317 16:18:38.445360 20088 connection.cc:232] RPC call timeout handler was delayed by -1.0090306269999019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
W20260317 16:18:38.503064 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 149531us
I20260317 16:18:38.455163 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.508605 20050 raft_consensus.cc:2468] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:38.477034 20045 raft_consensus.cc:2468] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:38.433739 21344 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260317 16:18:38.485374 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.510911 20045 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260317 16:18:38.492763 20041 raft_consensus.cc:2393] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:38.478504 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.438385 21340 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:38.523607 20088 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.514878 20120 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:38.453045 20116 raft_consensus.cc:1240] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:38.494699 21407 raft_consensus.cc:2764] T b444cc548a6940e98033a9a247164c51 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260317 16:18:38.535223 20045 raft_consensus.cc:3060] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:38.529531 21489 raft_consensus.cc:493] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.538517 21489 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.507736 21452 consensus_queue.cc:260] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
@ 0x7fea36422d25 testing::TestSuite::Run() at ??:0
I20260317 16:18:38.540156 21452 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:38.543421 20050 raft_consensus.cc:1240] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:38.541069 21489 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.545747 21216 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.478634 21325 raft_consensus.cc:515] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.476253 21432 raft_consensus.cc:2804] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:38.521409 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:38.543320 21154 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.475247 21479 raft_consensus.cc:697] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:38.564592 21489 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:38.565281 21489 raft_consensus.cc:493] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:38.567134 21489 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:38.254428 20047 rpcz_store.cc:276] 0317 16:18:35.957227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.958704 (+ 1477us) service_pool.cc:225] Handling call
0317 16:18:36.826255 (+867551us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:36.826468 (+ 213us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:36.826712 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:36.826751 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:36.827252 (+ 501us) log.cc:844] Serialized 26 byte log entry
0317 16:18:37.299285 (+472033us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:37.331559 (+ 32274us) log.cc:844] Serialized 14 byte log entry
0317 16:18:37.341149 (+ 9590us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:37.341234 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:37.855366 (+514132us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x7b440002304c. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36161065 00007fea36161a82 00007fea38eab57b 00007fea38ea6e4f 00007fea38e9ac0e 00007fea38e99f56 00007fea38e99b78 00007fea38ea35c1 00007fea38ea3579 00007fea38ea3509
0317 16:18:38.197403 (+342037us) raft_consensus.cc:1685] finished
0317 16:18:38.197432 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.197559 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16143,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":3474,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"spinlock_wait_cycles":22897280,"thread_start_us":391281,"threads_started":1,"wal-append.queue_time_us":829237}
I20260317 16:18:38.526494 21362 raft_consensus.cc:2804] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:38.572533 21519 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.572232 21262 consensus_queue.cc:237] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.574474 21423 leader_election.cc:290] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.575556 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:38.575769 21472 consensus_queue.cc:1048] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:38.576435 20044 raft_consensus.cc:3060] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 5
I20260317 16:18:38.577876 21211 consensus_queue.cc:1048] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:18:38.583575 20088 leader_election.cc:343] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:38.584043 21479 consensus_queue.cc:237] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.477276 21280 raft_consensus.cc:515] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.587343 21211 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.588511 21211 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.590237 21406 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:38.549441 21412 consensus_queue.cc:1059] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260317 16:18:38.591589 21211 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.592517 21425 leader_election.cc:290] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:38.549023 21480 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.525478 21340 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:38.584594 20088 leader_election.cc:343] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
I20260317 16:18:38.504403 21491 consensus_queue.cc:1059] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260317 16:18:38.613320 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 172856us
W20260317 16:18:38.628067 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 43509.4us
I20260317 16:18:38.550602 21432 raft_consensus.cc:493] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:38.633415 20088 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.596848 21406 raft_consensus.cc:3055] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Stepping down as leader of term 4
W20260317 16:18:38.478986 20042 rpcz_store.cc:269] 0317 16:18:35.913659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:35.926140 (+ 12481us) service_pool.cc:225] Handling call
0317 16:18:37.365337 (+1439197us) spinlock_profiling.cc:243] Waited 8.98 ms on lock 0x7b5800004598. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:38.363417 (+998080us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b54001b4ed4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:38.387841 (+ 24424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1604720640}
I20260317 16:18:38.643462 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 207) took 1998 ms. Trace:
I20260317 16:18:38.642407 20236 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:38.630474 21425 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:38.610703 20269 raft_consensus.cc:2468] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
I20260317 16:18:38.610953 21280 leader_election.cc:290] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.612354 21362 raft_consensus.cc:697] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:38.641672 21432 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:38.660248 20236 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.652935 20049 rpcz_store.cc:276] 0317 16:18:36.645528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:37.305988 (+660460us) service_pool.cc:225] Handling call
0317 16:18:37.321517 (+ 15529us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:37.325682 (+ 4165us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:38.185776 (+860094us) log.cc:844] Serialized 14 byte log entry
0317 16:18:38.625353 (+439577us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:38.625432 (+ 79us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:38.643117 (+ 17685us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.643226 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.643398 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17556,"prepare.queue_time_us":26640,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":2705,"raft.queue_time_us":34513,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":2629,"thread_start_us":1249885,"threads_started":3,"wal-append.queue_time_us":364138}
I20260317 16:18:38.649420 21406 raft_consensus.cc:740] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:38.633630 21480 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.657814 21471 raft_consensus.cc:2764] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
W20260317 16:18:38.680972 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 324641us
I20260317 16:18:38.668572 21473 raft_consensus.cc:2804] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:38.715420 21473 raft_consensus.cc:697] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:38.716641 21473 consensus_queue.cc:237] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.657366 21425 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.697922 21325 leader_election.cc:290] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.699936 20044 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
W20260317 16:18:38.701052 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 205691us
W20260317 16:18:38.702047 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 261575us
W20260317 16:18:38.706313 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.708793 21556 raft_consensus.cc:2804] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:38.721482 21556 raft_consensus.cc:493] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:38.722011 21556 raft_consensus.cc:3060] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:38.725437 21216 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:38.700215 21467 raft_consensus.cc:515] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.726068 21216 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.728225 21467 leader_election.cc:290] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.720505 21425 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:38.729290 21216 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:38.729326 20272 raft_consensus.cc:1240] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 4. Current term is 5. Ops: []
W20260317 16:18:38.730814 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.732122 21325 consensus_queue.cc:1048] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:38.732448 20198 raft_consensus.cc:1240] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 LEADER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260317 16:18:38.733834 20016 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.736140 21519 consensus_queue.cc:1059] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:38.736869 21519 consensus_queue.cc:1243] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer 7a8150828b9c4db7b4cdcf2de9f35096 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260317 16:18:38.736384 20016 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.737337 21519 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:38.738221 20016 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.738941 21423 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:38.739661 21423 raft_consensus.cc:3060] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:38.739706 20016 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:38.741207 20239 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.741252 20016 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.743407 21519 raft_consensus.cc:2749] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:38.744133 21325 raft_consensus.cc:2749] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:38.745590 21425 raft_consensus.cc:2749] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:38.747903 21423 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:38.756557 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:38.758584 20191 raft_consensus.cc:2468] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
I20260317 16:18:38.753247 21491 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:38.770048 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.773767 20015 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
W20260317 16:18:38.774731 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 114621us
I20260317 16:18:38.775615 20236 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:38.757983 21280 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:38.757810 21407 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:38.784209 20236 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:38.790971 20117 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46070 (request call id 219) took 2341 ms (client timeout 1608 ms). Trace:
W20260317 16:18:38.791445 20117 rpcz_store.cc:269] 0317 16:18:36.449506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:36.553919 (+104413us) service_pool.cc:225] Handling call
0317 16:18:37.354736 (+800817us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2c0160 00007fea3e2bfa44 00007fea37845834 00007fea378457c2
0317 16:18:38.790649 (+1435913us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54006bbcd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:38.790879 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1720163328}
I20260317 16:18:38.749610 21406 consensus_queue.cc:260] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.808424 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.813314 20273 raft_consensus.cc:2393] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:38.784016 21563 raft_consensus.cc:493] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:38.815554 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 74410.6us
I20260317 16:18:38.817973 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.787853 21423 raft_consensus.cc:2804] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:38.822625 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.823904 20116 raft_consensus.cc:1275] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:38.823812 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:38.824347 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:38.825605 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.756174 21362 consensus_queue.cc:237] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.776448 21491 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:38.836715 20321 tablet_bootstrap.cc:492] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:38.829931 20016 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:38.814515 21216 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:38.808168 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.783707 21280 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.840193 20321 ts_tablet_manager.cc:1403] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.967s user 0.010s sys 0.013s
I20260317 16:18:38.829474 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.788583 21407 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.842519 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:38.832630 21546 log.cc:927] Time spent T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.163s user 0.000s sys 0.002s
I20260317 16:18:38.827720 21568 raft_consensus.cc:2804] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:38.848138 20016 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:38.834822 21406 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:38.827401 21423 raft_consensus.cc:493] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:38.859566 21568 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:38.833706 21563 raft_consensus.cc:515] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:38.813782 21555 consensus_queue.cc:1048] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260317 16:18:38.845578 21480 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:38.874989 20016 leader_election.cc:304] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:38.845829 20043 raft_consensus.cc:2468] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:38.839527 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.905452 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.871629 21556 raft_consensus.cc:515] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:38.863665 21505 raft_consensus.cc:2749] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:38.838474 21479 consensus_queue.cc:1048] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20260317 16:18:38.905668 21473 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:38.910136 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.872587 21325 raft_consensus.cc:2749] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 16:18:38.872280 21549 log.cc:927] Time spent T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20260317 16:18:38.926357 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:38.927484 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:38.956007 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 212) took 2307 ms. Trace:
W20260317 16:18:38.974313 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:38.901185 21423 raft_consensus.cc:3060] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:38.985271 21567 log.cc:927] Time spent T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
W20260317 16:18:38.914844 20270 consensus_meta.cc:287] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.598s user 0.007s sys 0.007s
I20260317 16:18:38.904687 21425 raft_consensus.cc:493] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:38.986136 21489 raft_consensus.cc:515] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:38.913942 20269 raft_consensus.cc:2393] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:38.986327 21280 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:39.000413 21473 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:39.084465 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.085268 20271 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:38.955974 20266 raft_consensus.cc:2468] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:39.020921 20190 raft_consensus.cc:1275] T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:39.087397 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:39.087617 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.067467 21570 raft_consensus.cc:2749] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:39.086324 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.020910 21563 leader_election.cc:290] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
W20260317 16:18:39.088424 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.090605 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.091053 21425 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:39.065438 21407 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:39.093268 21425 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:38.908447 21519 raft_consensus.cc:2804] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:38.911808 21555 consensus_queue.cc:1048] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.171s
I20260317 16:18:39.085310 21473 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:38.904498 21568 raft_consensus.cc:3060] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:39.097610 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.086186 20048 rpcz_store.cc:276] 0317 16:18:36.648389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.528232 (+1879843us) service_pool.cc:225] Handling call
0317 16:18:38.528753 (+ 521us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:38.529693 (+ 940us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:38.652783 (+123090us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b18002b3b40. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36234d17 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea3624f3e5 00007fea3cd025f7 00007fea3930a8d7 00007fea39319fef 00007fea39319f68
0317 16:18:38.707481 (+ 54698us) log.cc:844] Serialized 14 byte log entry
0317 16:18:38.800615 (+ 93134us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:38.812154 (+ 11539us) log.cc:844] Serialized 26 byte log entry
0317 16:18:38.812346 (+ 192us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:38.812394 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.812491 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:38.913349 (+100858us) raft_consensus.cc:1685] finished
0317 16:18:38.955739 (+ 42390us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:38.955937 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8723,"prepare.queue_time_us":226205,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2647,"raft.queue_time_us":3331,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":20258,"spinlock_wait_cycles":55400704,"thread_start_us":96758,"threads_started":2,"wal-append.queue_time_us":92349}
I20260317 16:18:39.067289 21557 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:39.090087 21489 leader_election.cc:290] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:39.140507 20165 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:39.121860 20091 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.123.196:43213 timed out after 1.639s (SENT) [suppressed 42 similar messages]
W20260317 16:18:39.173461 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 51667.6us
W20260317 16:18:39.046519 20265 consensus_meta.cc:287] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.626s user 0.006s sys 0.005s
I20260317 16:18:38.956391 21479 consensus_queue.cc:1243] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer 795c49e9595f45bdb8d6b568498abf1f log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260317 16:18:39.164453 20321 raft_consensus.cc:359] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:39.094946 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.163962 21280 consensus_queue.cc:1048] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260317 16:18:39.169771 20270 raft_consensus.cc:2410] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260317 16:18:39.170053 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.171052 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.175546 20165 leader_election.cc:304] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:39.141820 21491 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:39.175335 21557 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:39.178673 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:39.183871 20321 raft_consensus.cc:385] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:39.188113 20321 raft_consensus.cc:740] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:39.099418 21519 raft_consensus.cc:493] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:39.188767 20321 consensus_queue.cc:260] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:39.184484 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 309583us
I20260317 16:18:39.202277 20120 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:39.175556 21407 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260317 16:18:39.224957 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.226435 20118 raft_consensus.cc:1240] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:39.226728 21552 consensus_queue.cc:1048] T e98310d917124048971895e5cd5c71f1 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20260317 16:18:39.227442 20196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 342) took 1060 ms. Trace:
I20260317 16:18:39.227742 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.227813 20196 rpcz_store.cc:276] 0317 16:18:38.167225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.240101 (+ 72876us) service_pool.cc:225] Handling call
0317 16:18:38.240393 (+ 292us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:38.240732 (+ 339us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:38.240758 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:38.241194 (+ 436us) log.cc:844] Serialized 26 byte log entry
0317 16:18:38.358243 (+117049us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:38.358280 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.358349 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:39.227255 (+868906us) raft_consensus.cc:1685] finished
0317 16:18:39.227274 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:39.227381 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":221,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":116180,"threads_started":1,"wal-append.queue_time_us":391739}
I20260317 16:18:39.229001 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.229645 20118 raft_consensus.cc:2468] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:39.230259 20267 raft_consensus.cc:1240] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:39.231326 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:39.231956 20120 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:39.233582 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.235178 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:39.240760 20163 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.236712 21556 leader_election.cc:290] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:39.237283 21570 consensus_queue.cc:1048] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:39.203243 21519 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:39.238971 21280 consensus_queue.cc:1059] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:39.217535 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.179744 20265 raft_consensus.cc:2468] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
W20260317 16:18:39.274608 21340 consensus_meta.cc:287] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.632s user 0.007s sys 0.006s
I20260317 16:18:39.275132 21340 raft_consensus.cc:515] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:39.276741 20198 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:39.278328 21280 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:39.262629 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:39.274688 21432 consensus_meta.cc:287] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.603s user 0.004s sys 0.008s
I20260317 16:18:39.281286 21340 leader_election.cc:290] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:39.284998 21557 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:39.242367 21407 raft_consensus.cc:493] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:39.242311 21579 raft_consensus.cc:2804] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:39.266561 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:39.301966 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 206220us
I20260317 16:18:39.270300 21504 raft_consensus.cc:2764] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
W20260317 16:18:39.297065 21561 log.cc:927] Time spent T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.506s user 0.005s sys 0.005s
W20260317 16:18:39.302901 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:39.291445 20265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34240 (request call id 214) took 1019 ms. Trace:
I20260317 16:18:39.301667 21407 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:39.302234 21579 raft_consensus.cc:493] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:39.317989 20120 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:39.319343 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.307716 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:39.299332 21432 raft_consensus.cc:515] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:39.298276 21280 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:39.312546 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.328619 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.308288 20265 rpcz_store.cc:276] 0317 16:18:38.272057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.297604 (+ 25547us) service_pool.cc:225] Handling call
0317 16:18:39.271417 (+973813us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:39.336216 20268 raft_consensus.cc:2393] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 3: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:39.318710 21579 raft_consensus.cc:3060] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:39.337459 20321 ts_tablet_manager.cc:1434] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.479s user 0.019s sys 0.018s
I20260317 16:18:39.338594 21493 raft_consensus.cc:3055] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:39.340696 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.347311 20125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 281) took 1114 ms. Trace:
I20260317 16:18:39.328354 20271 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:39.349095 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.354259 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:39.346891 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.355451 20120 raft_consensus.cc:2468] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:39.353120 20125 rpcz_store.cc:276] 0317 16:18:38.232475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.255898 (+ 23423us) service_pool.cc:225] Handling call
0317 16:18:38.288113 (+ 32215us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:38.289177 (+ 1064us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:38.289222 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:38.289813 (+ 591us) log.cc:844] Serialized 26 byte log entry
0317 16:18:38.479169 (+189356us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:38.691490 (+212321us) log.cc:844] Serialized 14 byte log entry
0317 16:18:38.691894 (+ 404us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:38.692138 (+ 244us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:38.692197 (+ 59us) raft_consensus.cc:1685] finished
0317 16:18:38.692208 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:39.347161 (+654953us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b54003cee50. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea392ff7dd 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:39.347263 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":301230,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":2081,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":11850,"spinlock_wait_cycles":73248640,"thread_start_us":372631,"threads_started":2,"wal-append.queue_time_us":251833}
W20260317 16:18:39.351198 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 159799us
I20260317 16:18:39.368623 20321 tablet_bootstrap.cc:492] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
I20260317 16:18:39.329921 21423 raft_consensus.cc:515] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:39.370048 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 965477us
I20260317 16:18:39.369601 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.375166 20238 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:39.376890 20016 leader_election.cc:304] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:39.351912 21493 raft_consensus.cc:740] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:39.403877 21493 consensus_queue.cc:260] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:39.378016 20264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 256) took 1144 ms. Trace:
I20260317 16:18:39.381438 21470 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:39.381698 21587 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:39.387300 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.387655 21523 raft_consensus.cc:3055] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Stepping down as leader of term 4
W20260317 16:18:39.421304 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 94297.8us
I20260317 16:18:39.359597 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:39.378326 21280 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:39.397325 21432 leader_election.cc:290] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:39.434342 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:39.398453 21580 log.cc:927] Time spent T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260317 16:18:39.399353 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:39.397418 20272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56776 (request call id 246) took 1226 ms. Trace:
I20260317 16:18:39.434214 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:39.417505 20120 raft_consensus.cc:1275] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260317 16:18:39.420018 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:39.437336 20272 rpcz_store.cc:276] 0317 16:18:38.170971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.172480 (+ 1509us) service_pool.cc:225] Handling call
0317 16:18:38.181233 (+ 8753us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:38.730209 (+548976us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:39.397175 (+666966us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54006847d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:39.397359 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1260039936}
I20260317 16:18:39.436235 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.405740 21493 raft_consensus.cc:3060] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:39.438153 20199 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:39.438808 20273 raft_consensus.cc:2468] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:39.443102 20121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 280) took 1914 ms. Trace:
I20260317 16:18:39.446946 20195 raft_consensus.cc:1240] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:39.374298 21582 consensus_queue.cc:1059] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260317 16:18:39.474203 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.431077 21468 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:39.475179 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:39.475639 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.475984 20197 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:39.476873 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.457551 20121 rpcz_store.cc:276] 0317 16:18:37.529020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.216630 (+687610us) service_pool.cc:225] Handling call
0317 16:18:38.217010 (+ 380us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:38.236508 (+ 19498us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:38.236542 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:38.236594 (+ 52us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:38.236628 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:39.442830 (+1206202us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b54001f90d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:39.442893 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:39.443044 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44345088}
I20260317 16:18:39.481479 20271 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 6.
I20260317 16:18:39.482287 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.483510 20273 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 6.
I20260317 16:18:39.453195 20190 raft_consensus.cc:1275] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:39.485869 21458 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:39.483131 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.440232 20265 raft_consensus.cc:1240] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:39.450105 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:39.478318 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "629f02fa987f4ce2a3b1522ee599490b" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.429136 21523 raft_consensus.cc:740] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:39.482333 20321 tablet_bootstrap.cc:654] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
W20260317 16:18:39.466971 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.497092 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.504793 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:39.506045 20196 raft_consensus.cc:2393] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 6: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:39.508006 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:39.527449 20272 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:39.431911 21587 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:39.486855 21586 consensus_queue.cc:1048] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.152s
I20260317 16:18:39.493183 21523 consensus_queue.cc:260] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:39.702721 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:39.638329 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 262303us
I20260317 16:18:39.442615 20117 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:39.700683 20197 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
W20260317 16:18:39.701206 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 268871us
I20260317 16:18:39.541103 20199 raft_consensus.cc:1240] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:39.713032 21423 leader_election.cc:290] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:39.560632 21467 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:39.716969 21467 raft_consensus.cc:493] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:39.717361 21467 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:39.418025 20264 rpcz_store.cc:276] 0317 16:18:38.233429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.235879 (+ 2450us) service_pool.cc:225] Handling call
0317 16:18:38.236188 (+ 309us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:38.310425 (+ 74237us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:39.364398 (+1053973us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b540028ebd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:39.375039 (+ 10641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51826560}
I20260317 16:18:39.725617 21523 raft_consensus.cc:3060] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:39.710197 21458 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:39.721520 21407 raft_consensus.cc:515] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:39.647616 21519 raft_consensus.cc:515] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:39.740859 21519 leader_election.cc:290] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:39.736637 21474 raft_consensus.cc:493] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:39.692159 21580 log.cc:927] Time spent T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260317 16:18:39.733028 21458 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:39.715121 21568 consensus_meta.cc:287] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.512s user 0.006s sys 0.010s
I20260317 16:18:39.746685 20194 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: []
W20260317 16:18:39.747501 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.777060 21568 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:39.739347 20016 leader_election.cc:304] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:39.690101 21593 raft_consensus.cc:2749] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:39.750592 21587 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:39.748088 20196 raft_consensus.cc:1275] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:39.751251 21407 leader_election.cc:290] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:39.750450 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:39.782323 21586 raft_consensus.cc:2749] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:39.782250 20198 raft_consensus.cc:3060] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:39.784785 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 47828.9us
I20260317 16:18:39.752811 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:39.581552 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:39.753961 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.742441 20238 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:39.620718 21344 consensus_queue.cc:1048] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.213s
I20260317 16:18:39.776222 21458 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:39.751632 20091 leader_election.cc:304] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:39.787376 21586 consensus_queue.cc:1048] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:39.792483 20016 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:39.800037 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 57679.8us
W20260317 16:18:39.826869 20165 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.834692 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.830488 20165 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:39.839689 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 47314.1us
I20260317 16:18:39.833671 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:39.841289 20272 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:39.842180 20016 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:39.854905 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:39.851514 20162 leader_election.cc:304] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:39.860455 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 119043us
I20260317 16:18:39.861846 20124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 294) took 1038 ms. Trace:
W20260317 16:18:39.863142 20163 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.863234 20162 leader_election.cc:304] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:39.862247 20124 rpcz_store.cc:276] 0317 16:18:38.823845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.832157 (+ 8312us) service_pool.cc:225] Handling call
0317 16:18:38.835332 (+ 3175us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:38.835625 (+ 293us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:38.835648 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:38.842741 (+ 7093us) log.cc:844] Serialized 26 byte log entry
0317 16:18:39.182484 (+339743us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:39.205117 (+ 22633us) log.cc:844] Serialized 14 byte log entry
0317 16:18:39.218015 (+ 12898us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:39.218088 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:39.861637 (+643549us) raft_consensus.cc:1685] finished
0317 16:18:39.861659 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:39.861789 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":35073,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":15304,"raft.queue_time_us":920,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":5106,"thread_start_us":281614,"threads_started":1,"wal-append.queue_time_us":439083}
I20260317 16:18:39.871519 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.872697 20271 raft_consensus.cc:2393] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:39.878304 20267 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:39.840396 21470 raft_consensus.cc:2804] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:39.845389 21412 raft_consensus.cc:2804] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Leader election won for term 6
I20260317 16:18:39.791862 21474 raft_consensus.cc:515] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:39.885761 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:39.848250 21552 raft_consensus.cc:2749] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:39.889030 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:39.859313 21519 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260317 16:18:39.913570 20267 raft_consensus.cc:3060] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:39.870417 21458 raft_consensus.cc:2804] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
@ 0x7fea36437815 testing::internal::UnitTestImpl::RunAllTests() at ??:0
I20260317 16:18:39.945801 21458 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:39.946264 21458 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:39.879403 21582 raft_consensus.cc:493] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:39.947527 21582 raft_consensus.cc:515] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:39.949826 20265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:39.950177 21582 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:39.859977 21333 raft_consensus.cc:493] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:39.871377 21472 raft_consensus.cc:2804] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 16:18:39.908169 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:39.985486 20271 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 6: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:39.868523 21452 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:39.993069 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 131908us
I20260317 16:18:39.920902 21568 leader_election.cc:290] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:39.927135 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.928043 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:39.932328 20268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 183) took 1115 ms. Trace:
I20260317 16:18:39.879212 21587 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:39.934906 21412 raft_consensus.cc:697] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
W20260317 16:18:39.987499 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:39.943030 21574 consensus_queue.cc:1059] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260317 16:18:39.944756 20090 connection.cc:232] RPC call timeout handler was delayed by -1.0047652369999014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
I20260317 16:18:40.017822 20090 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:39.945334 21474 leader_election.cc:290] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:39.879801 21593 raft_consensus.cc:493] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:39.879462 21407 raft_consensus.cc:2804] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:39.951845 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:39.877235 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:39.993269 21333 raft_consensus.cc:515] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:39.954072 21560 log.cc:927] Time spent T 629f02fa987f4ce2a3b1522ee599490b P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 1.164s user 0.000s sys 0.024s
I20260317 16:18:40.038873 20264 raft_consensus.cc:2468] T a9f666d8f908427cb4523d57a45eb2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:39.953959 21470 raft_consensus.cc:697] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:40.054725 20048 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:40.054901 21470 consensus_queue.cc:237] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.055992 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.056553 20273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:40.056968 20048 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:40.057555 20273 raft_consensus.cc:2393] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 2: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:40.057976 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:39.945066 21154 raft_consensus.cc:3055] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:40.058789 20048 raft_consensus.cc:2393] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 6: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
W20260317 16:18:39.958223 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet b444cc548a6940e98033a9a247164c51 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet e9d151fbf3e044cc96cd724bdd41c1b3
I20260317 16:18:40.059216 20042 raft_consensus.cc:2393] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:40.060204 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:39.956635 21579 consensus_meta.cc:287] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.601s user 0.000s sys 0.013s
W20260317 16:18:40.061738 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet aa7ec7cb7aad43e8950899557d4022f7 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 732a4ec897b2469a9a4fe3e41c5d8997
W20260317 16:18:39.959741 20043 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 243) took 1772 ms (client timeout 1639 ms). Trace:
I20260317 16:18:39.962720 20119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46070 (request call id 225) took 2464 ms. Trace:
I20260317 16:18:39.963699 20266 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:40.065780 20119 rpcz_store.cc:276] 0317 16:18:37.498416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.170826 (+672410us) service_pool.cc:225] Handling call
0317 16:18:38.179268 (+ 8442us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:38.710436 (+531168us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:38.710481 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:38.710539 (+ 58us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:38.710576 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:39.962523 (+1251947us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54002e3454. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:39.962565 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:39.962668 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1866,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":40160,"spinlock_wait_cycles":1220683136}
I20260317 16:18:40.066817 20266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34248 (request call id 192) took 1245 ms. Trace:
I20260317 16:18:40.067268 20266 rpcz_store.cc:276] 0317 16:18:38.820775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:39.087138 (+266363us) service_pool.cc:225] Handling call
0317 16:18:40.066354 (+979216us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b5400394254. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:40.066574 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2012746496}
I20260317 16:18:39.966372 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e5551ca32124ae686a779b9eafe976e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.068647 21333 leader_election.cc:290] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:40.043869 21407 raft_consensus.cc:697] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:40.070024 20044 raft_consensus.cc:2393] T 7e5551ca32124ae686a779b9eafe976e P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 6: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:39.885591 20193 raft_consensus.cc:2410] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260317 16:18:39.968631 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.070981 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.050573 20047 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:40.071962 20044 raft_consensus.cc:2368] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6.
I20260317 16:18:39.879312 21280 consensus_queue.cc:1048] T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.088s
I20260317 16:18:39.901538 20198 raft_consensus.cc:2468] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:40.072172 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:39.998749 20268 rpcz_store.cc:276] 0317 16:18:38.817392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.817590 (+ 198us) service_pool.cc:225] Handling call
0317 16:18:39.931951 (+1114361us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400225e54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:39.932219 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1169845504}
I20260317 16:18:40.073024 20041 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:39.971091 20194 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:40.000684 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:39.937305 21594 log.cc:927] Time spent T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260317 16:18:39.993312 21472 raft_consensus.cc:493] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:40.015231 21587 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.083544 21472 raft_consensus.cc:3060] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:40.087656 20116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 296) took 1129 ms. Trace:
I20260317 16:18:39.995455 21572 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:39.953325 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.089665 20050 raft_consensus.cc:3060] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:40.053299 20267 raft_consensus.cc:1275] T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:40.058636 21154 raft_consensus.cc:740] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
W20260317 16:18:40.062964 20043 rpcz_store.cc:269] 0317 16:18:38.187085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.224514 (+ 37429us) service_pool.cc:225] Handling call
0317 16:18:39.959539 (+1735025us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54000393d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:39.959670 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":568204672}
I20260317 16:18:40.062070 21579 raft_consensus.cc:515] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.064826 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:40.058631 21586 raft_consensus.cc:493] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.088191 20116 rpcz_store.cc:276] 0317 16:18:38.957787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.987723 (+ 29936us) service_pool.cc:225] Handling call
0317 16:18:39.557906 (+570183us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800018398. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea36214d12 00007fea3f0ccf89 00007fea3e249330 00007fea3e31234b 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:39.572964 (+ 15058us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:39.584639 (+ 11675us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:39.585322 (+ 683us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:39.586942 (+ 1620us) log.cc:844] Serialized 26 byte log entry
0317 16:18:39.790185 (+203243us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:39.790228 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:39.790325 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:40.072725 (+282400us) raft_consensus.cc:1685] finished
0317 16:18:40.072746 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:40.087390 (+ 14644us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2320,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":135,"spinlock_wait_cycles":31646592,"thread_start_us":202523,"threads_started":1,"wal-append.queue_time_us":412214}
I20260317 16:18:39.988201 21467 raft_consensus.cc:515] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.074277 20271 raft_consensus.cc:2410] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260317 16:18:39.997171 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 146665us
I20260317 16:18:40.070667 20272 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
W20260317 16:18:40.092971 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 225234us
W20260317 16:18:40.094933 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 218999us
I20260317 16:18:40.096109 20015 leader_election.cc:304] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:40.101449 20269 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:39.998468 21452 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.105383 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:40.072240 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7f7f224bb064944a05175b00d835fce" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.109140 20162 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.116407 20321 tablet_bootstrap.cc:492] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:40.117115 20321 ts_tablet_manager.cc:1403] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.760s user 0.051s sys 0.040s
I20260317 16:18:40.110036 21242 raft_consensus.cc:3055] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:40.039444 21593 raft_consensus.cc:515] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:40.118602 21216 raft_consensus.cc:2804] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:40.119015 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.120440 21601 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:40.153834 20198 raft_consensus.cc:1275] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:40.147224 21483 raft_consensus.cc:493] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.143847 21280 raft_consensus.cc:2804] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:40.156086 21280 raft_consensus.cc:697] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:40.155992 21483 raft_consensus.cc:515] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:40.157572 21280 consensus_queue.cc:237] T 021523da58cc4c16babb94f5b4b5cee8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.159601 21483 leader_election.cc:290] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
W20260317 16:18:40.163762 21600 log.cc:927] Time spent T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260317 16:18:40.133149 21154 consensus_queue.cc:260] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.165438 21587 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:40.165690 21154 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:40.178625 21404 raft_consensus.cc:3060] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:40.126809 20046 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 213) took 1930 ms (client timeout 2195 ms). Trace:
I20260317 16:18:40.101639 21572 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:40.183178 20046 rpcz_store.cc:269] 0317 16:18:38.194595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:38.472567 (+277972us) service_pool.cc:225] Handling call
0317 16:18:40.101381 (+1628814us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540049cfd4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea3930422a 00007fea3e2bfbd7 00007fea37845834 00007fea378457c2 00007fea378456f1 00007fea3784566c 00007fea37844922 00007fea3763f34c 00007fea3763e786 00007fea3763e12f
0317 16:18:40.124516 (+ 23135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3144576896}
I20260317 16:18:40.184842 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.185460 20046 raft_consensus.cc:2376] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1. Re-sending same reply.
I20260317 16:18:40.186380 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.119472 21519 consensus_queue.cc:1059] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260317 16:18:40.166611 20044 raft_consensus.cc:2468] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:40.187775 21467 leader_election.cc:290] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:40.188309 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.188992 20044 raft_consensus.cc:3060] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:40.191072 20042 raft_consensus.cc:2410] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260317 16:18:40.152700 21555 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.192183 21555 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:40.129325 21242 raft_consensus.cc:740] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:40.111586 21586 raft_consensus.cc:515] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.135845 20265 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260317 16:18:40.173792 21423 raft_consensus.cc:3055] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:40.144578 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.177942 21579 leader_election.cc:290] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:40.147593 21407 consensus_queue.cc:237] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.197781 20050 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:40.127245 20047 raft_consensus.cc:2468] T b7f7f224bb064944a05175b00d835fce P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 4.
W20260317 16:18:40.200259 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 4dfc403ac89c4d339adb26bfd789d1a2 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 0d1bf93668494e60916699f830c277c7
W20260317 16:18:40.210314 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 192610us
I20260317 16:18:40.213440 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.115345 21412 consensus_queue.cc:237] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:40.226138 20163 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.152525 21216 raft_consensus.cc:697] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Becoming Leader. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:40.250980 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.258011 20048 raft_consensus.cc:2468] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:40.258962 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.260236 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.260773 20048 raft_consensus.cc:2393] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:40.255430 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.261684 20050 raft_consensus.cc:2376] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5. Re-sending same reply.
I20260317 16:18:40.255477 20090 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:40.210871 21423 raft_consensus.cc:740] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:40.282284 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:40.307869 21597 log.cc:927] Time spent T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260317 16:18:40.308320 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:40.309041 20195 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:40.309370 21593 leader_election.cc:290] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:40.315413 21216 consensus_queue.cc:237] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.323829 21452 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:40.323896 21586 leader_election.cc:290] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:40.324527 21452 raft_consensus.cc:493] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:40.224573 21566 consensus_queue.cc:1048] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20260317 16:18:40.325212 21555 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:40.325647 21242 consensus_queue.cc:260] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.326808 21242 raft_consensus.cc:3060] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:40.328075 20321 raft_consensus.cc:359] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.328781 20321 raft_consensus.cc:385] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:40.329031 20321 raft_consensus.cc:740] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:40.329666 20321 consensus_queue.cc:260] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.216413 20046 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 6.
I20260317 16:18:40.334486 20321 ts_tablet_manager.cc:1434] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.217s user 0.010s sys 0.000s
I20260317 16:18:40.325034 21452 raft_consensus.cc:515] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.354573 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.357170 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.357801 20050 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:40.356191 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:40.369402 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 270516us
I20260317 16:18:40.379369 20196 raft_consensus.cc:1240] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:40.381583 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:40.385260 20195 raft_consensus.cc:2468] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 6.
W20260317 16:18:40.396728 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:40.370059 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:40.421797 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.406873 20165 leader_election.cc:304] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
W20260317 16:18:40.391216 21600 log.cc:927] Time spent T d7faa93fffc94522a88d5c47721186bd P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260317 16:18:40.437923 20321 tablet_bootstrap.cc:492] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b: Bootstrap starting.
W20260317 16:18:40.460198 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 466216us
W20260317 16:18:40.455288 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:40.470032 20238 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:40.369117 21605 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:40.456336 20050 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
W20260317 16:18:40.473711 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 218307us
I20260317 16:18:40.477625 20090 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:40.489648 20321 tablet_bootstrap.cc:654] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b: Neither blocks nor log segments found. Creating new log.
I20260317 16:18:40.390462 21617 raft_consensus.cc:493] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
W20260317 16:18:40.495596 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet d7faa93fffc94522a88d5c47721186bd (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet b0830eef30554c3688ad0522457cb59c
W20260317 16:18:40.436280 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.437266 21340 raft_consensus.cc:2764] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260317 16:18:40.502658 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 273919us
I20260317 16:18:40.324219 21423 consensus_queue.cc:260] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.527930 21423 raft_consensus.cc:3060] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:40.529140 21488 raft_consensus.cc:493] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.528925 21333 consensus_queue.cc:1059] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:40.531101 21333 raft_consensus.cc:2804] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:40.530845 21488 raft_consensus.cc:515] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.531510 21333 raft_consensus.cc:493] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:40.531898 21333 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:40.533245 21488 leader_election.cc:290] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:40.534703 21608 log.cc:927] Time spent T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260317 16:18:40.505077 21617 raft_consensus.cc:515] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.535233 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:40.535981 20195 raft_consensus.cc:2393] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:40.537022 21617 leader_election.cc:290] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:40.537330 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9f666d8f908427cb4523d57a45eb2b3" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.495157 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.538308 20046 raft_consensus.cc:3060] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:40.467108 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:40.557687 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 46469us
W20260317 16:18:40.549305 20162 leader_election.cc:336] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.876s (SENT)
I20260317 16:18:40.538652 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.458549 21458 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.557015 20041 raft_consensus.cc:2468] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 0.
I20260317 16:18:40.592276 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:40.588994 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 231490us
I20260317 16:18:40.509115 21574 raft_consensus.cc:2804] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader election won for term 6
I20260317 16:18:40.603248 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.510051 21613 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:40.611980 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.614105 20041 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:40.615310 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:40.615446 21612 log.cc:927] Time spent T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260317 16:18:40.616075 20123 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46046 (request call id 298) took 1439 ms. Trace:
I20260317 16:18:40.597445 20045 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:40.616466 20123 rpcz_store.cc:276] 0317 16:18:39.176368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:39.176524 (+ 156us) service_pool.cc:225] Handling call
0317 16:18:39.176958 (+ 434us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:39.185909 (+ 8951us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:39.185946 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:39.186373 (+ 427us) log.cc:844] Serialized 26 byte log entry
0317 16:18:40.161494 (+975121us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b18003abac0. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36234d17 00007fea3f028d39 00007fea3625162b 00007fea3624f8cf 00007fea36251b92 00007fea38e995ab 00007fea38e9f0c3 00007fea38e9f471 00007fea392bb93f
0317 16:18:40.437964 (+276470us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:40.438023 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:40.438131 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:40.615769 (+177638us) raft_consensus.cc:1685] finished
0317 16:18:40.615790 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:40.616015 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5861,"raft.queue_time_us":24699,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":20565,"spinlock_wait_cycles":36285696,"thread_start_us":276227,"threads_started":1,"wal-append.queue_time_us":1389806}
I20260317 16:18:40.618064 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.619488 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.620426 20045 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:40.623143 20198 raft_consensus.cc:1240] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:40.623334 20041 raft_consensus.cc:2468] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:40.623904 21626 raft_consensus.cc:2804] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:40.624414 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:40.625800 20041 raft_consensus.cc:2468] T fd3be07aef26463ba2eaa9c72c6fa094 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 1.
I20260317 16:18:40.624826 21626 raft_consensus.cc:697] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [term 2 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:40.621987 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.629314 20045 raft_consensus.cc:3060] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:40.630033 21623 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.555034 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.544606 21452 leader_election.cc:290] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:40.622255 21458 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:40.609757 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:40.646412 20162 leader_election.cc:336] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.770s (SENT)
I20260317 16:18:40.547055 21340 consensus_queue.cc:1059] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260317 16:18:40.548383 21572 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:40.673596 20269 raft_consensus.cc:1275] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:40.676260 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:40.677366 20270 raft_consensus.cc:2393] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 5: Already voted for candidate d8ed4b12dc654776aa7e64c0fa224321 in this term.
I20260317 16:18:40.674858 20162 leader_election.cc:304] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:40.658527 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:40.593253 20047 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
I20260317 16:18:40.693073 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:40.676584 20041 raft_consensus.cc:1240] T 49737061a2c940b9982aa52ccdf66e90 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:40.494290 21605 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.681824 21458 raft_consensus.cc:3055] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:40.553244 21474 consensus_queue.cc:1059] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.095s
I20260317 16:18:40.683244 21606 raft_consensus.cc:2804] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 16:18:40.684329 20162 leader_election.cc:304] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.723124 20045 raft_consensus.cc:2468] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 1.
I20260317 16:18:40.651130 21574 raft_consensus.cc:697] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 6 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:40.701433 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 264826us
I20260317 16:18:40.541122 21472 raft_consensus.cc:515] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:40.723935 20162 leader_election.cc:304] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:40.691095 20236 leader_election.cc:336] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.602s (SENT)
I20260317 16:18:40.812330 20165 leader_election.cc:304] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:40.812942 20236 leader_election.cc:304] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:40.686719 21404 consensus_meta.cc:287] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096: Time spent flushing consensus metadata: real 0.507s user 0.006s sys 0.004s
I20260317 16:18:40.829504 21404 raft_consensus.cc:2804] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260317 16:18:40.709342 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.829823 21280 raft_consensus.cc:2749] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 16:18:40.829965 21404 raft_consensus.cc:493] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:40.687613 20044 raft_consensus.cc:2468] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:40.831166 21277 raft_consensus.cc:493] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.831772 21277 raft_consensus.cc:515] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:40.833459 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 142405us
I20260317 16:18:40.833834 20044 raft_consensus.cc:1275] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:40.689990 21333 raft_consensus.cc:515] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.827922 21472 leader_election.cc:290] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
W20260317 16:18:40.828763 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 106004us
I20260317 16:18:40.726471 21606 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:40.613935 21613 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.830437 21404 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:40.834527 21488 raft_consensus.cc:2749] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260317 16:18:40.855546 21599 log.cc:927] Time spent T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20260317 16:18:40.700320 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.845497 20236 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:40.848881 21242 consensus_meta.cc:287] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.522s user 0.008s sys 0.000s
W20260317 16:18:40.849938 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 112204us
I20260317 16:18:40.870622 20198 raft_consensus.cc:1275] T d4f0e7842cc94d539a8537a2b131711b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:40.872804 20236 leader_election.cc:304] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:40.821388 20046 raft_consensus.cc:2468] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:40.709467 21458 raft_consensus.cc:740] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:40.829325 20165 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:40.878746 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 49526us
W20260317 16:18:40.879155 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 416577us
I20260317 16:18:40.879794 20165 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.858938 20162 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:40.855010 21224 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:40.860448 21277 leader_election.cc:290] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:40.856020 21566 consensus_queue.cc:1059] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260317 16:18:40.896844 20162 leader_election.cc:304] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.848901 21628 raft_consensus.cc:3055] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:40.861140 21333 leader_election.cc:290] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:40.863204 21586 raft_consensus.cc:493] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:40.868547 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 391042us
I20260317 16:18:40.835536 20193 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:40.874202 20041 raft_consensus.cc:2376] T d3bc11389a01493f9af4639ba9d9e449 P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3. Re-sending same reply.
I20260317 16:18:40.881326 21242 raft_consensus.cc:2749] T fd397002573f4a5ba2091161eaea1a47 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260317 16:18:40.877017 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.868093 21606 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Advancing to term 4
I20260317 16:18:40.729190 21572 raft_consensus.cc:2749] T d8bd34d6b1ae4fe5bf58a857d0f837ea P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:40.890977 21224 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.892050 21605 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:40.882088 21607 raft_consensus.cc:2749] T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260317 16:18:40.929726 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.902612 20165 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.898496 20162 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.900053 20091 leader_election.cc:304] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:40.878042 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.958269 21242 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:40.878679 21601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.009s sys 0.024s
I20260317 16:18:40.958887 21242 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:40.959542 21601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.009s sys 0.024s
I20260317 16:18:40.959844 21601 raft_consensus.cc:697] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:40.961129 21242 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:40.908732 20090 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.961988 21452 raft_consensus.cc:2804] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260317 16:18:40.962103 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 53468.4us
I20260317 16:18:40.963037 21627 raft_consensus.cc:2749] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:40.963385 20090 leader_election.cc:400] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885) with higher term. Message: Invalid argument: T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:40.963711 20090 leader_election.cc:403] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:40.900621 21566 raft_consensus.cc:3055] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 16:18:40.964267 21566 raft_consensus.cc:740] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:40.965754 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37f744805bd5488a8790301a2537fef1" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:40.966288 20123 raft_consensus.cc:2376] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5. Re-sending same reply.
I20260317 16:18:40.966441 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:40.967120 20194 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:40.913137 20041 raft_consensus.cc:1240] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:40.968619 20041 raft_consensus.cc:1240] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
W20260317 16:18:40.969043 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:40.871623 20016 leader_election.cc:304] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.969992 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:40.970583 20123 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
W20260317 16:18:40.896056 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:40.962041 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 62092.3us
I20260317 16:18:40.876087 21216 consensus_queue.cc:1048] T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.033s
W20260317 16:18:40.917596 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 321788us
W20260317 16:18:40.973198 20088 leader_election.cc:336] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.598s (SENT)
I20260317 16:18:40.877316 21467 consensus_queue.cc:1048] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260317 16:18:40.973698 20238 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:40.890482 21154 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:40.894821 21633 raft_consensus.cc:2804] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:40.906801 21280 raft_consensus.cc:2804] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:40.974925 21627 raft_consensus.cc:2749] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885) with higher term. Message: Invalid argument: T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:40.975191 21280 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:40.975813 21280 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:40.975739 21627 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:40.977653 21627 raft_consensus.cc:493] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:40.978432 20048 raft_consensus.cc:2468] T fd397002573f4a5ba2091161eaea1a47 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 6.
I20260317 16:18:40.904099 21628 raft_consensus.cc:740] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:40.942975 21412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.028s sys 0.014s
I20260317 16:18:40.962673 21452 raft_consensus.cc:493] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:40.973743 20088 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.979292 21452 raft_consensus.cc:3060] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:40.974731 21154 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:40.980361 20088 leader_election.cc:336] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.631s (SENT)
I20260317 16:18:40.981120 21154 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:40.974973 21467 raft_consensus.cc:2804] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:40.982124 21467 raft_consensus.cc:493] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:40.982450 21467 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:40.979035 21412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.028s sys 0.014s
I20260317 16:18:40.980876 20088 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:40.985294 20088 leader_election.cc:336] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.651s (SENT)
I20260317 16:18:40.974709 21633 raft_consensus.cc:493] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:40.908346 21611 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:40.985855 20088 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:40.908188 21579 raft_consensus.cc:2749] T 4c55cdb999654153a4cb90e34cb77217 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:40.986465 21633 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:40.975751 20091 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:40.990501 21452 raft_consensus.cc:515] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:40.991984 21467 raft_consensus.cc:515] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:40.894778 21458 consensus_queue.cc:260] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:40.924086 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
@ 0x7fea36443cf0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
W20260317 16:18:40.995431 21588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.021s sys 0.021s
W20260317 16:18:40.999444 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet c963e7aa6d46472691e7e0de834008ab (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet cdaf4c5f7c7347e5a934a53001fc9f2a
I20260317 16:18:41.000905 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:40.907665 21617 consensus_queue.cc:1059] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
W20260317 16:18:41.008724 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 106216us
I20260317 16:18:41.013513 20165 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:41.009676 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 111291us
W20260317 16:18:41.010910 21588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.021s sys 0.021s
I20260317 16:18:41.016436 21154 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 16:18:41.034305 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 162772us
I20260317 16:18:40.917130 21586 raft_consensus.cc:515] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.035181 20165 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.049286 20165 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.027643 21611 raft_consensus.cc:2804] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:40.978092 21627 raft_consensus.cc:515] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:40.650410 21623 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.032016 21216 raft_consensus.cc:493] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:41.065305 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet f26f14718ce8437ab76d99156bfa1fa3 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 5de4fd1c9c3b46faa38346546bb8b6c9
I20260317 16:18:40.860936 21574 consensus_queue.cc:237] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.062299 20165 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.073432 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 295) took 1192 ms. Trace:
I20260317 16:18:40.960788 21601 consensus_queue.cc:237] T 6d13fb540668437094ca4190aa5dd7ec P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.071262 20048 raft_consensus.cc:1240] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [term 2 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:40.965610 21566 consensus_queue.cc:260] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.037073 21407 raft_consensus.cc:2749] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:41.047381 21636 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:40.862965 21626 consensus_queue.cc:237] T 34a510b634dc4a808c9f7af78ba4cd7e P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.084847 20049 rpcz_store.cc:276] 0317 16:18:39.880653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:39.886671 (+ 6018us) service_pool.cc:225] Handling call
0317 16:18:39.929279 (+ 42608us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:39.972078 (+ 42799us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:39.972123 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:39.972982 (+ 859us) log.cc:844] Serialized 26 byte log entry
0317 16:18:40.201805 (+228823us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:40.497610 (+295805us) log.cc:844] Serialized 14 byte log entry
0317 16:18:40.499043 (+ 1433us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.003042 (+503999us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b1800146320. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea36220386 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9
0317 16:18:41.072995 (+ 69953us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b54001b3ad4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:41.073065 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:41.073129 (+ 64us) raft_consensus.cc:1685] finished
0317 16:18:41.073148 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.073348 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":103888,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":37,"raft.queue_time_us":2157,"raft.run_cpu_time_us":566,"raft.run_wall_time_us":23742,"spinlock_wait_cycles":113050752,"thread_start_us":442221,"threads_started":2,"wal-append.queue_time_us":417739}
I20260317 16:18:41.067857 21572 raft_consensus.cc:2804] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:41.060604 21458 raft_consensus.cc:3060] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:41.060132 21340 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:41.093255 20048 raft_consensus.cc:1240] T 83e3771d4d494b8caa5d647d8073666c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Rejecting Update request from peer 7a8150828b9c4db7b4cdcf2de9f35096 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:41.066273 20044 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 308) took 1166 ms. Trace:
I20260317 16:18:41.060945 21611 raft_consensus.cc:493] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:41.051362 21523 raft_consensus.cc:2749] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:41.082883 20046 raft_consensus.cc:3060] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:41.085548 21613 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.092029 21617 raft_consensus.cc:2749] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 16:18:41.109035 21566 raft_consensus.cc:3060] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:41.109489 21472 raft_consensus.cc:3055] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Stepping down as leader of term 4
W20260317 16:18:41.115435 21638 log.cc:927] Time spent T d7faa93fffc94522a88d5c47721186bd P 9a3afb098d6a45f0b405566771a1157b: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260317 16:18:41.089805 21579 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:40.984005 21628 consensus_queue.cc:260] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.132997 21606 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.060474 21637 raft_consensus.cc:2749] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260317 16:18:41.086114 20043 consensus_meta.cc:287] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.832s user 0.013s sys 0.000s
I20260317 16:18:41.087291 20165 leader_election.cc:304] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.064168 21452 leader_election.cc:290] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.068022 21588 consensus_queue.cc:1048] T b444cc548a6940e98033a9a247164c51 P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260317 16:18:41.085692 21216 raft_consensus.cc:515] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.081872 21635 raft_consensus.cc:2749] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:41.171684 21613 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
W20260317 16:18:41.180433 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 3a2894e5c7f64210ba697507a2823f16 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 4bacade8681b46a4ba9fc15d6fdd803a
W20260317 16:18:41.194660 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 316676us
I20260317 16:18:41.153301 21611 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:41.187198 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:41.187300 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.174729 21617 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:41.138517 21572 raft_consensus.cc:493] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:41.170660 21472 raft_consensus.cc:740] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:41.197846 20049 raft_consensus.cc:1275] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20260317 16:18:41.192314 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 105115us
I20260317 16:18:41.190496 21280 raft_consensus.cc:515] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.205062 21216 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.197165 21224 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.258363 21523 consensus_queue.cc:1059] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:41.258697 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:41.258898 20046 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 253) took 1245 ms. Trace:
I20260317 16:18:41.213135 21340 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.259289 21472 consensus_queue.cc:260] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.259894 21340 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.260746 21472 raft_consensus.cc:3060] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:41.233350 20165 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:41.262198 21340 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:41.220348 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.221565 20190 raft_consensus.cc:1275] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 16:18:41.264652 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.257095 21280 leader_election.cc:290] T 41e4ff5d2d614a199e8e8a314b43bb7c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.221660 21404 raft_consensus.cc:515] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.224877 21613 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:41.266916 20042 consensus_meta.cc:287] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b: Time spent flushing consensus metadata: real 0.576s user 0.011s sys 0.000s
I20260317 16:18:41.267346 20042 raft_consensus.cc:2468] T 37f744805bd5488a8790301a2537fef1 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:41.267623 21613 leader_election.cc:290] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.267940 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56408 (request call id 239) took 1109 ms. Trace:
I20260317 16:18:41.268337 20042 rpcz_store.cc:276] 0317 16:18:40.158421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.554646 (+396225us) service_pool.cc:225] Handling call
0317 16:18:41.267872 (+713226us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:41.268684 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.226040 20197 raft_consensus.cc:1240] T fd397002573f4a5ba2091161eaea1a47 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 LEADER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: []
W20260317 16:18:41.269390 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 091c367bf7d84c61b2a329d98d708cc8 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 73e8dc0b63b2433cb61da533ccd373de
I20260317 16:18:41.259398 20193 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:41.228773 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 356026us
I20260317 16:18:41.271963 20236 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
I20260317 16:18:41.273741 20236 leader_election.cc:304] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:41.232471 20049 pending_rounds.cc:85] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Aborting all ops after (but not including) 2
I20260317 16:18:41.259342 20046 rpcz_store.cc:276] 0317 16:18:40.013455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.893638 (+880183us) service_pool.cc:225] Handling call
0317 16:18:40.894034 (+ 396us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:41.230744 (+336710us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.258644 (+ 27900us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400394c54. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:41.258843 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":421655296}
I20260317 16:18:41.274950 20049 pending_rounds.cc:107] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260317 16:18:41.275296 20049 raft_consensus.cc:2889] T a07475a709344f5e812eea08aa4113cd P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:41.276700 21242 raft_consensus.cc:2804] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:41.277063 20049 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 236) took 1119 ms. Trace:
I20260317 16:18:41.277302 21242 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:41.277860 21242 raft_consensus.cc:3060] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:41.277607 20049 rpcz_store.cc:276] 0317 16:18:40.157484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:41.107551 (+950067us) service_pool.cc:225] Handling call
0317 16:18:41.111043 (+ 3492us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:41.275982 (+164939us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.276789 (+ 807us) spinlock_profiling.cc:243] Waited 86.4 ms on lock 0x7b540003a7d4. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea392697a1 00007fea39300d31 00007fea392ff76e 00007fea3e2be4d9 00007fea37840d34 00007fea37840cc2 00007fea37840bf1 00007fea37840b6c 00007fea3783fe22 00007fea3763f34c 00007fea3763e786
0317 16:18:41.276997 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198646528}
I20260317 16:18:41.280175 21488 raft_consensus.cc:2804] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:41.280910 21488 raft_consensus.cc:493] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:41.281324 21488 raft_consensus.cc:3060] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Advancing to term 2
W20260317 16:18:41.280323 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet fb3cfd29098d4ab9a179a3d07ebb75ee (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 81b52fecdefc49c68a188bf361b79338
I20260317 16:18:41.195737 21579 raft_consensus.cc:2749] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:41.283057 21425 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.235544 21617 raft_consensus.cc:515] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.283880 21425 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.232052 21628 raft_consensus.cc:3060] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:41.226440 21607 raft_consensus.cc:493] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.191341 20043 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:41.233752 21572 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:41.287189 20043 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56392 (request call id 264) took 1185 ms. Trace:
I20260317 16:18:41.147421 20044 rpcz_store.cc:276] 0317 16:18:40.005075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.877236 (+872161us) service_pool.cc:225] Handling call
0317 16:18:40.951042 (+ 73806us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:41.014401 (+ 63359us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:41.017044 (+ 2643us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:41.028635 (+ 11591us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:41.029982 (+ 1347us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.037137 (+ 7155us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.061178 (+ 24041us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 16:18:41.287809 20043 rpcz_store.cc:276] 0317 16:18:40.101758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.121688 (+ 19930us) service_pool.cc:225] Handling call
0317 16:18:41.287111 (+1165423us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 16:18:41.301096 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 81baa6e6b5864f69b8da5c3418aea90a (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet ef793f59542946028f149073787e536f
I20260317 16:18:41.194983 21467 leader_election.cc:290] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.360136 20321 tablet_bootstrap.cc:492] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b: No bootstrap required, opened a new log
I20260317 16:18:41.360920 20321 ts_tablet_manager.cc:1403] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b: Time spent bootstrapping tablet: real 0.997s user 0.070s sys 0.070s
I20260317 16:18:41.270401 21412 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260317 16:18:41.380512 20199 raft_consensus.cc:1240] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:41.383262 20041 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 262) took 1308 ms. Trace:
I20260317 16:18:41.385251 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.386067 20197 raft_consensus.cc:2393] T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:41.348654 21470 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:41.392907 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.348510 21623 leader_election.cc:290] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.384277 21586 leader_election.cc:290] T 65896e94a12d4953b9bbfff15ac73867 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
W20260317 16:18:41.448086 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.234190 21633 raft_consensus.cc:515] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.352631 21607 raft_consensus.cc:515] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.453329 21607 leader_election.cc:290] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.376026 21593 consensus_queue.cc:1048] T fb3cfd29098d4ab9a179a3d07ebb75ee P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.079s
W20260317 16:18:41.385874 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet fd3be07aef26463ba2eaa9c72c6fa094 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 0655eca7a62246cfbee966a152eec78d
I20260317 16:18:41.403393 20190 raft_consensus.cc:1240] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:41.421614 21523 raft_consensus.cc:493] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:41.464283 21523 raft_consensus.cc:515] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
W20260317 16:18:41.464993 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 34a510b634dc4a808c9f7af78ba4cd7e (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 50ea3086951d4fdd9ee1b616a0b5e96f
W20260317 16:18:41.467089 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.468387 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.471184 21493 consensus_queue.cc:1059] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:41.468575 21617 leader_election.cc:290] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.473860 21523 leader_election.cc:290] T 4c55cdb999654153a4cb90e34cb77217 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.384727 20041 rpcz_store.cc:276] 0317 16:18:40.074926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.969263 (+894337us) service_pool.cc:225] Handling call
0317 16:18:40.969500 (+ 237us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:40.969700 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 1
0317 16:18:40.969936 (+ 236us) log.cc:844] Serialized 14 byte log entry
0317 16:18:41.382792 (+412856us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:41.382868 (+ 76us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:41.382930 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.383029 (+ 99us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.383190 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4199,"prepare.queue_time_us":752,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":5554,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":14172,"thread_start_us":408189,"threads_started":1}
I20260317 16:18:41.424021 21606 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:41.425746 21404 leader_election.cc:290] T c902369cf6954cb695208719bf353017 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.429162 21627 leader_election.cc:290] T a663a988cc754c24b227bb9bc85540db P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.431756 21425 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.435335 21470 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:41.436658 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.481405 21470 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.491856 21633 leader_election.cc:290] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.496817 21611 raft_consensus.cc:515] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.580025 21611 leader_election.cc:290] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.589075 20321 raft_consensus.cc:359] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.589988 20321 raft_consensus.cc:385] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 16:18:41.590303 20321 raft_consensus.cc:740] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Initialized, Role: FOLLOWER
I20260317 16:18:41.591450 20321 consensus_queue.cc:260] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.593890 20321 ts_tablet_manager.cc:1434] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b: Time spent starting tablet: real 0.233s user 0.006s sys 0.001s
W20260317 16:18:41.618662 20045 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.000s sys 0.024s
W20260317 16:18:41.551077 20091 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.016s sys 0.030s
W20260317 16:18:41.621016 20091 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.017s sys 0.030s
W20260317 16:18:41.621618 20091 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 645899us
I20260317 16:18:41.622388 20091 leader_election.cc:304] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:41.623281 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 419561us
I20260317 16:18:41.623574 20091 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:41.624580 20091 leader_election.cc:304] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.625327 21657 raft_consensus.cc:2749] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:41.626148 21654 raft_consensus.cc:2804] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260317 16:18:41.626235 21657 raft_consensus.cc:2749] T 629f02fa987f4ce2a3b1522ee599490b P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:41.626686 21654 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 16:18:41.560045 20016 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.008s sys 0.011s
I20260317 16:18:41.627121 21654 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Advancing to term 7
W20260317 16:18:41.560716 21583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.007s sys 0.011s
W20260317 16:18:41.627637 21583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.007s sys 0.011s
I20260317 16:18:41.627225 21657 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260317 16:18:41.561239 20198 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.011s sys 0.006s
I20260317 16:18:41.562898 21652 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:41.628654 20198 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s user 0.011s sys 0.007s
W20260317 16:18:41.563786 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.628830 21652 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:41.627207 20016 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.009s sys 0.011s
W20260317 16:18:41.630134 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.196:43213, user_credentials={real_user=slave}} blocked reactor thread for 594207us
I20260317 16:18:41.630909 20016 leader_election.cc:304] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.632643 21588 raft_consensus.cc:2749] T a29e5bafc1d14cf4932e2a0489026dd2 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260317 16:18:41.564894 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:41.566650 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:41.634146 20237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.634258 20048 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 247) took 1113 ms. Trace:
W20260317 16:18:41.630863 21602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.022s sys 0.124s
W20260317 16:18:41.635083 21602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s user 0.023s sys 0.124s
W20260317 16:18:41.562909 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 204826us
I20260317 16:18:41.553197 21423 raft_consensus.cc:3055] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:41.636803 21423 raft_consensus.cc:740] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9a3afb098d6a45f0b405566771a1157b, State: Running, Role: LEADER
I20260317 16:18:41.637595 21423 consensus_queue.cc:260] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.571300 20192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55456 (request call id 377) took 1203 ms. Trace:
I20260317 16:18:41.575610 21242 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:41.576695 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 343445us
I20260317 16:18:41.576870 21606 raft_consensus.cc:2804] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260317 16:18:41.537375 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 65896e94a12d4953b9bbfff15ac73867 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 8cd4816b884843a99ccd503f26516a7a
I20260317 16:18:41.638706 21602 raft_consensus.cc:3055] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:41.639827 21602 raft_consensus.cc:740] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:41.640381 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet c198f5201a944b26bd4d2f870953e117 (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet ec8c708cd91f4c629ac05a02e9c624b6
I20260317 16:18:41.640712 21602 consensus_queue.cc:260] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.641980 21602 raft_consensus.cc:3060] T 7e5551ca32124ae686a779b9eafe976e P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:41.639339 21606 raft_consensus.cc:493] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:41.643373 21606 raft_consensus.cc:3060] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:41.639449 20165 leader_election.cc:304] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:41.645673 21579 raft_consensus.cc:2804] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 16:18:41.645792 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.646178 21579 raft_consensus.cc:697] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:41.646262 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.647226 20190 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2.
I20260317 16:18:41.647334 21579 consensus_queue.cc:237] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.651371 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.543545 21572 raft_consensus.cc:515] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.655511 20191 raft_consensus.cc:1240] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 4. Current term is 5. Ops: []
I20260317 16:18:41.638944 20192 rpcz_store.cc:276] 0317 16:18:40.368408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.376215 (+ 7807us) service_pool.cc:225] Handling call
0317 16:18:40.839843 (+463628us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b58001b9f98. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea36092911 00007fea36214e61 00007fea362151de 000000000035a3c9 00007fea3e249630 00007fea3e312364 00007fea3e3123d1 00007fea3e2d4080 00007fea3e2be9e9 00007fea3e2be492 00007fea37840d34
0317 16:18:40.840294 (+ 451us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:40.871653 (+ 31359us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:40.880758 (+ 9105us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:40.891482 (+ 10724us) log.cc:844] Serialized 26 byte log entry
0317 16:18:41.190661 (+299179us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:41.196518 (+ 5857us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.570752 (+374234us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:41.570870 (+ 118us) raft_consensus.cc:1685] finished
0317 16:18:41.570889 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.571220 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17069,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"spinlock_wait_cycles":46938368,"thread_start_us":285495,"threads_started":1,"wal-append.queue_time_us":519381}
W20260317 16:18:41.620528 20088 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.004s sys 0.009s
W20260317 16:18:41.686591 20088 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.005s sys 0.009s
I20260317 16:18:41.692664 20194 raft_consensus.cc:1240] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 2. Current term is 3. Ops: []
I20260317 16:18:41.634617 20048 rpcz_store.cc:276] 0317 16:18:40.521083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:41.147133 (+626050us) service_pool.cc:225] Handling call
0317 16:18:41.147526 (+ 393us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:41.147882 (+ 356us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:41.147907 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:41.148366 (+ 459us) log.cc:844] Serialized 26 byte log entry
0317 16:18:41.633918 (+485552us) raft_consensus.cc:1645] Marking committed up to 0
0317 16:18:41.633959 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.634019 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:41.634061 (+ 42us) raft_consensus.cc:1685] finished
0317 16:18:41.634073 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.634190 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22156,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2285,"thread_start_us":484909,"threads_started":1,"wal-append.queue_time_us":404481}
I20260317 16:18:41.728037 21423 raft_consensus.cc:3060] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:41.731253 20164 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.661660 20193 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:41.732379 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.733068 20041 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 6.
I20260317 16:18:41.663905 20047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 260) took 1627 ms. Trace:
I20260317 16:18:41.664001 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f0e7842cc94d539a8537a2b131711b" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.734130 20123 raft_consensus.cc:3060] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:41.733887 20047 rpcz_store.cc:276] 0317 16:18:40.037075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.958464 (+921389us) service_pool.cc:225] Handling call
0317 16:18:40.958917 (+ 453us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:41.016802 (+ 57885us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:41.026634 (+ 9832us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:41.078929 (+ 52295us) log.cc:844] Serialized 26 byte log entry
0317 16:18:41.615804 (+536875us) raft_consensus.cc:1645] Marking committed up to 2
0317 16:18:41.616499 (+ 695us) log.cc:844] Serialized 14 byte log entry
0317 16:18:41.616944 (+ 445us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.617048 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:41.663646 (+ 46598us) raft_consensus.cc:1685] finished
0317 16:18:41.663676 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.663845 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37299,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":68954,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":5459,"thread_start_us":492343,"threads_started":1,"wal-append.queue_time_us":492778}
W20260317 16:18:41.736949 20002 catalog_manager.cc:6134] T 00000000000000000000000000000000 P aebfd8ce7b084fe1af5511a23409311f: Tablet 7ef41a8cfc184979b0e3fc5d3b370bcd (table test-workload [id=b34906c5f0e84d5b916910674fc7e084]) was not created within the allowed timeout. Replacing with a new tablet 5e65ab0d10824e19a4f1f8721b7a92d6
I20260317 16:18:41.667234 20122 raft_consensus.cc:1240] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:41.737663 21611 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:41.738101 21611 raft_consensus.cc:493] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:41.738551 21611 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:41.668259 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.739491 20191 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5.
I20260317 16:18:41.671375 21467 consensus_queue.cc:1059] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:41.742919 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ef41a8cfc184979b0e3fc5d3b370bcd" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.743822 20041 raft_consensus.cc:2393] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:41.744771 21652 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.742506 21467 raft_consensus.cc:3055] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Stepping down as leader of term 5
W20260317 16:18:41.747123 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 101412us
W20260317 16:18:41.674404 20162 leader_election.cc:336] T 37f744805bd5488a8790301a2537fef1 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 2.118s (SENT)
W20260317 16:18:41.738123 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 281105us
W20260317 16:18:41.748157 20162 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 73790.1us
I20260317 16:18:41.750406 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.751365 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.751629 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83e3771d4d494b8caa5d647d8073666c" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:41.751751 21611 raft_consensus.cc:515] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.753185 20197 raft_consensus.cc:1240] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 4. Ops: []
I20260317 16:18:41.753772 21611 leader_election.cc:290] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.731117 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.764511 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.764768 20043 raft_consensus.cc:2468] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 0.
I20260317 16:18:41.765165 20125 raft_consensus.cc:2468] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 6.
I20260317 16:18:41.765219 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.660168 21572 leader_election.cc:290] T edd06ee443fc4680b217c7dcbc1f640a P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.765913 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.549636 21488 raft_consensus.cc:515] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.733551 20116 raft_consensus.cc:1275] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260317 16:18:41.767163 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b444cc548a6940e98033a9a247164c51" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.768060 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.768678 20117 raft_consensus.cc:3060] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 3
I20260317 16:18:41.768831 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.770119 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.666199 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.747200 21467 raft_consensus.cc:740] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:41.770655 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.771468 20043 raft_consensus.cc:1240] T 4ff46c37d04d446981ad8728f73c6e1f P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:41.771183 21467 consensus_queue.cc:260] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.770087 20047 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:41.755306 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
W20260317 16:18:41.775961 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.756911 20123 raft_consensus.cc:2468] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:41.777493 20125 raft_consensus.cc:1275] T d4f0e7842cc94d539a8537a2b131711b P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:41.777995 20162 leader_election.cc:304] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:41.778273 20199 raft_consensus.cc:2468] T 83e3771d4d494b8caa5d647d8073666c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 3.
W20260317 16:18:41.759320 21574 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.007s sys 0.020s
I20260317 16:18:41.778728 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.779081 21611 raft_consensus.cc:2804] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:41.777518 20047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.779361 20195 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 5.
I20260317 16:18:41.779862 21611 raft_consensus.cc:697] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:41.779943 20047 raft_consensus.cc:3060] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:41.780754 21611 consensus_queue.cc:237] T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.782359 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1234018d42ac449782301c404118e5bb" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.784648 20194 raft_consensus.cc:1240] T fb3cfd29098d4ab9a179a3d07ebb75ee P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:41.786206 20265 raft_consensus.cc:1275] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260317 16:18:41.787067 20117 raft_consensus.cc:2468] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:41.760175 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.787432 21242 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:41.788025 20041 raft_consensus.cc:2393] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
W20260317 16:18:41.759596 21592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.000s sys 0.026s
I20260317 16:18:41.761502 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
@ 0x7fea36436dcd testing::UnitTest::Run() at ??:0
I20260317 16:18:41.788794 20049 raft_consensus.cc:2376] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 5. Re-sending same reply.
I20260317 16:18:41.788918 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.762917 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.789665 20196 raft_consensus.cc:2368] T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
@ 0x378fbc RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:?
I20260317 16:18:41.789800 20044 raft_consensus.cc:2376] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 2. Re-sending same reply.
@ 0x377b8d main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:?
I20260317 16:18:41.763075 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.763935 20046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.764068 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5875427d66a54f92898d04d79d1bc6de" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.790413 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.791314 20046 raft_consensus.cc:2393] T 60cb13d3aec54fd58f4a4821692ab2b3 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 5: Already voted for candidate 9a3afb098d6a45f0b405566771a1157b in this term.
I20260317 16:18:41.791368 20122 raft_consensus.cc:2393] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:41.772383 21467 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:41.792157 21592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.000s sys 0.027s
W20260317 16:18:41.793287 20050 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.018s sys 0.008s
W20260317 16:18:41.793620 20050 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s user 0.018s sys 0.008s
I20260317 16:18:41.794301 20050 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56392 (request call id 263) took 1707 ms. Trace:
I20260317 16:18:41.749040 20191 raft_consensus.cc:1217] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
W20260317 16:18:41.779158 21574 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s user 0.008s sys 0.020s
I20260317 16:18:41.795109 20191 raft_consensus.cc:1240] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 16:18:41.773360 21488 leader_election.cc:290] T fd3be07aef26463ba2eaa9c72c6fa094 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.738682 20042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56408 (request call id 249) took 1143 ms. Trace:
I20260317 16:18:41.796289 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.796414 20042 rpcz_store.cc:276] 0317 16:18:40.595708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:41.269247 (+673539us) service_pool.cc:225] Handling call
0317 16:18:41.269584 (+ 337us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:41.269820 (+ 236us) raft_consensus.cc:1534] Early marking committed up to index 4
0317 16:18:41.595347 (+325527us) log.cc:844] Serialized 14 byte log entry
0317 16:18:41.738337 (+142990us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:41.738405 (+ 68us) raft_consensus.cc:1645] Marking committed up to 4
0317 16:18:41.738451 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.738516 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.738621 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":316907,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":1052,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":84,"thread_start_us":142602,"threads_started":1,"wal-append.queue_time_us":139544}
I20260317 16:18:41.775112 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.798456 21242 raft_consensus.cc:493] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.799177 20236 leader_election.cc:304] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:41.799067 21242 raft_consensus.cc:515] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.799500 20162 leader_election.cc:304] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:41.799921 21566 raft_consensus.cc:493] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.800400 21572 raft_consensus.cc:2804] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:41.800383 21566 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.801101 21242 leader_election.cc:290] T d8bd34d6b1ae4fe5bf58a857d0f837ea P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:41.801491 21572 raft_consensus.cc:697] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:41.802611 21566 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:41.802791 21672 raft_consensus.cc:2804] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:41.802493 21572 consensus_queue.cc:237] T d3bc11389a01493f9af4639ba9d9e449 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.803299 21672 raft_consensus.cc:697] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:41.804301 21672 consensus_queue.cc:237] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
W20260317 16:18:41.619299 21333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.005s sys 0.018s
W20260317 16:18:41.806054 21333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.006s sys 0.018s
I20260317 16:18:41.787112 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:41.811344 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.811664 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.812564 20122 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 4.
I20260317 16:18:41.802234 21242 consensus_queue.cc:1059] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260317 16:18:41.815120 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.815616 21566 raft_consensus.cc:3055] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:41.816076 21566 raft_consensus.cc:740] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:41.816879 21566 consensus_queue.cc:260] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:41.817852 21566 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:41.818389 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dfc403ac89c4d339adb26bfd789d1a2" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.820901 20122 raft_consensus.cc:3055] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:41.821194 20122 raft_consensus.cc:740] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:41.821866 20122 consensus_queue.cc:260] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.822835 20122 raft_consensus.cc:3060] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:41.760316 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.674669 20238 leader_election.cc:304] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
W20260317 16:18:41.824486 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 150201us
I20260317 16:18:41.825663 21488 raft_consensus.cc:2804] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:41.826023 21488 raft_consensus.cc:493] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:41.826361 21488 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
W20260317 16:18:41.678753 20045 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.000s sys 0.024s
I20260317 16:18:41.826933 21470 consensus_queue.cc:1059] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260317 16:18:41.827325 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56386 (request call id 305) took 1933 ms. Trace:
I20260317 16:18:41.827737 20045 rpcz_store.cc:276] 0317 16:18:39.893650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:40.828075 (+934425us) service_pool.cc:225] Handling call
0317 16:18:40.828406 (+ 331us) raft_consensus.cc:1471] Updating replica for 0 ops
0317 16:18:40.828604 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 3
0317 16:18:40.828908 (+ 304us) log.cc:844] Serialized 14 byte log entry
0317 16:18:41.067266 (+238358us) log.cc:844] Serialized 14 byte log entry
0317 16:18:41.097508 (+ 30242us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b180037a620. stack: 00007fea36220423 00007fea36220320 00007fea35b72574 00007fea3f83a6b4 00007fea3f83a649 00007fea3f0b4001 00007fea3f0c0861 00007fea361c74ca 00007fea3f8595c9 00007fea360871c0 00007fea3624f474 00007fea3624f3e5 00007fea392f7f90 00007fea3930a646 00007fea3931b1b7 00007fea3931b138
0317 16:18:41.826976 (+729468us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0317 16:18:41.827035 (+ 59us) raft_consensus.cc:1645] Marking committed up to 3
0317 16:18:41.827085 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.827153 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.827272 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27866,"prepare.queue_time_us":1872,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":71,"raft.queue_time_us":142960,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":171,"spinlock_wait_cycles":4475776,"thread_start_us":929632,"threads_started":2,"wal-append.queue_time_us":436529}
I20260317 16:18:41.829031 21470 consensus_queue.cc:1059] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260317 16:18:41.829847 21607 raft_consensus.cc:3055] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:41.830184 21607 raft_consensus.cc:740] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:41.759182 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:41.787226 20190 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260317 16:18:41.838462 21673 raft_consensus.cc:3055] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Stepping down as leader of term 2
W20260317 16:18:41.838920 20239 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.794649 20050 rpcz_store.cc:276] 0317 16:18:40.087034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 16:18:41.012157 (+925123us) service_pool.cc:225] Handling call
0317 16:18:41.031697 (+ 19540us) raft_consensus.cc:1471] Updating replica for 1 ops
0317 16:18:41.062996 (+ 31299us) raft_consensus.cc:1534] Early marking committed up to index 0
0317 16:18:41.063044 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0317 16:18:41.070640 (+ 7596us) log.cc:844] Serialized 26 byte log entry
0317 16:18:41.793947 (+723307us) raft_consensus.cc:1645] Marking committed up to 1
0317 16:18:41.793994 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 16:18:41.794068 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 16:18:41.794109 (+ 41us) raft_consensus.cc:1685] finished
0317 16:18:41.794120 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 16:18:41.794250 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30888,"raft.queue_time_us":92067,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":62,"thread_start_us":674662,"threads_started":1,"wal-append.queue_time_us":654979}
I20260317 16:18:41.826514 20238 leader_election.cc:304] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:41.785334 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43d7dcb7b21548889a850700415701a9" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.777792 21606 raft_consensus.cc:515] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:41.733073 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 747778us
I20260317 16:18:41.835321 21607 consensus_queue.cc:260] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:41.835875 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:41.851749 20088 leader_election.cc:336] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Timed out: RequestConsensusVote RPC to 127.16.123.193:40319 timed out after 1.812s (SENT)
I20260317 16:18:41.851857 20238 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:41.852157 20088 leader_election.cc:304] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.853083 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c55cdb999654153a4cb90e34cb77217" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:41.848834 20122 raft_consensus.cc:1275] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Refusing update from remote peer d8ed4b12dc654776aa7e64c0fa224321: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260317 16:18:41.853816 20122 pending_rounds.cc:85] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Aborting all ops after (but not including) 2
I20260317 16:18:41.854064 20122 pending_rounds.cc:107] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260317 16:18:41.854318 20122 raft_consensus.cc:2889] T 5875427d66a54f92898d04d79d1bc6de P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:41.851372 21607 raft_consensus.cc:3060] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:41.858561 21647 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:41.859077 21647 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:41.859412 21647 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:41.839020 21673 raft_consensus.cc:740] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:41.874828 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8fd5a883fbb4ae1bcc76cb626ccb7f2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:41.877067 20088 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.877921 20088 leader_election.cc:304] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:41.878559 20272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ead98604210744a2b26b15fb54b5ea29" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:41.880477 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bd34d6b1ae4fe5bf58a857d0f837ea" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.870028 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.870425 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3bc11389a01493f9af4639ba9d9e449" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:41.870492 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.882149 20123 raft_consensus.cc:2376] T d3bc11389a01493f9af4639ba9d9e449 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3. Re-sending same reply.
W20260317 16:18:41.882486 20091 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.882548 20122 raft_consensus.cc:2393] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 5: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:41.883384 20123 raft_consensus.cc:1275] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Refusing update from remote peer 7a8150828b9c4db7b4cdcf2de9f35096: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260317 16:18:41.883621 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60cb13d3aec54fd58f4a4821692ab2b3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.884359 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff46c37d04d446981ad8728f73c6e1f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.885758 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81baa6e6b5864f69b8da5c3418aea90a" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.888959 21658 raft_consensus.cc:493] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.889622 21658 raft_consensus.cc:515] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:41.890949 21652 consensus_queue.cc:1048] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:41.891693 21658 leader_election.cc:290] T 021523da58cc4c16babb94f5b4b5cee8 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:41.891898 20123 raft_consensus.cc:2393] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 1: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:41.892207 21658 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:41.892688 21658 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:41.892951 20123 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f26f14718ce8437ab76d99156bfa1fa3" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:41.895478 20165 connection.cc:232] RPC call timeout handler was delayed by -1.2130423329999758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2717s for a response
I20260317 16:18:41.929147 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:41.935053 20165 connection.cc:232] RPC call timeout handler was delayed by -1.1137844690001657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
I20260317 16:18:41.899559 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.935940 20045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41e4ff5d2d614a199e8e8a314b43bb7c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.936585 20045 raft_consensus.cc:3060] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:41.936388 20120 raft_consensus.cc:2468] T c963e7aa6d46472691e7e0de834008ab P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:41.900790 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.928290 20090 leader_election.cc:304] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:41.902237 20267 raft_consensus.cc:1240] T 7e5551ca32124ae686a779b9eafe976e P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:41.939515 20090 leader_election.cc:304] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:41.856848 21470 raft_consensus.cc:2804] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:41.967005 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.967212 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.967849 20116 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:41.968124 20125 raft_consensus.cc:2393] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a8150828b9c4db7b4cdcf2de9f35096 in current term 6: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:41.861160 21602 raft_consensus.cc:2749] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:41.934980 20123 raft_consensus.cc:2468] T f26f14718ce8437ab76d99156bfa1fa3 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 2.
I20260317 16:18:41.980652 20123 raft_consensus.cc:1240] T 37f744805bd5488a8790301a2537fef1 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260317 16:18:41.975883 20041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.941200 20195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:41.942238 20090 leader_election.cc:400] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885) with higher term. Message: Invalid argument: T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:41.982715 20090 leader_election.cc:403] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260317 16:18:41.982191 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
I20260317 16:18:41.942072 20198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.943189 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd06ee443fc4680b217c7dcbc1f640a" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.983234 20042 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:41.983678 20272 raft_consensus.cc:1240] T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:41.984189 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.985096 20199 raft_consensus.cc:2393] T fca192bf1e8848e2a2dac227f537c9e2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f in current term 2: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:41.985219 20116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.985826 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.985930 20044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.986860 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e6529a0cb514111b1bfd58537d255d2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:41.943526 20271 raft_consensus.cc:1240] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 1. Current term is 2. Ops: []
I20260317 16:18:41.988039 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0709d1e3fc7349f2b93ed3e602c1a723" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:41.988775 20271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca192bf1e8848e2a2dac227f537c9e2" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:41.989385 20271 raft_consensus.cc:2468] T fca192bf1e8848e2a2dac227f537c9e2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 2.
I20260317 16:18:41.990044 20268 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98310d917124048971895e5cd5c71f1" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:41.949283 20267 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c198f5201a944b26bd4d2f870953e117" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:41.990442 20049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c902369cf6954cb695208719bf353017" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.990700 20267 raft_consensus.cc:3060] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:41.991003 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c963e7aa6d46472691e7e0de834008ab" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.991258 20191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.991868 20273 raft_consensus.cc:1240] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 4. Current term is 5. Ops: []
W20260317 16:18:41.949621 19996 auto_rebalancer.cc:248] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260317 16:18:41.942154 20194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.993182 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "091c367bf7d84c61b2a329d98d708cc8" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:41.993520 20191 raft_consensus.cc:1240] T a29e5bafc1d14cf4932e2a0489026dd2 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:41.993790 20119 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:41.994158 20194 raft_consensus.cc:1240] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 4. Current term is 5. Ops: []
I20260317 16:18:41.940788 20197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa7ec7cb7aad43e8950899557d4022f7" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.947937 21601 raft_consensus.cc:2804] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:41.995390 20050 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80b48fd655d14d649c65e99261b01ca9" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b" is_pre_election: true
W20260317 16:18:41.972391 20238 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.996090 20050 raft_consensus.cc:2468] T 80b48fd655d14d649c65e99261b01ca9 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 2.
I20260317 16:18:41.973892 20165 leader_election.cc:304] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:41.975657 20196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd3be07aef26463ba2eaa9c72c6fa094" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:41.976621 20048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a29e5bafc1d14cf4932e2a0489026dd2" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:41.995625 20266 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
W20260317 16:18:41.998023 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 55918.1us
I20260317 16:18:41.999181 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:41.999946 20199 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:41.991034 20049 raft_consensus.cc:3060] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:41.990455 20271 raft_consensus.cc:1240] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [term 6 LEADER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260317 16:18:42.000663 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
W20260317 16:18:42.001144 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.010516 20165 leader_election.cc:304] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:41.981869 20041 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
W20260317 16:18:42.010542 20088 leader_election.cc:343] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319): Illegal state: must be running to vote when last-logged opid is not known
W20260317 16:18:42.024091 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.025373 21611 raft_consensus.cc:2804] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:42.039353 20042 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb492ec38d074e94994a3a2a9e245bac" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
W20260317 16:18:41.968120 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:41.894989 21587 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:42.043381 20090 leader_election.cc:304] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.033135 20122 raft_consensus.cc:2468] T 60cb13d3aec54fd58f4a4821692ab2b3 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
W20260317 16:18:41.982962 21686 log.cc:927] Time spent T 4ff46c37d04d446981ad8728f73c6e1f P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260317 16:18:42.041043 20125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.032146 20045 raft_consensus.cc:2468] T 41e4ff5d2d614a199e8e8a314b43bb7c P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:41.894596 21658 leader_election.cc:290] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:41.979059 20192 raft_consensus.cc:1240] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 3. Current term is 5. Ops: []
I20260317 16:18:41.966979 21472 raft_consensus.cc:2804] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election won for term 3
I20260317 16:18:42.047147 20015 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:41.977662 21623 raft_consensus.cc:2804] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 16:18:41.995573 21601 raft_consensus.cc:697] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:41.873890 21579 consensus_queue.cc:1048] T a07475a709344f5e812eea08aa4113cd P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
I20260317 16:18:42.067195 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:42.055035 20267 raft_consensus.cc:2468] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
I20260317 16:18:42.012225 21333 raft_consensus.cc:2749] T 0709d1e3fc7349f2b93ed3e602c1a723 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885) with higher term. Message: Invalid argument: T 0709d1e3fc7349f2b93ed3e602c1a723 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3.
I20260317 16:18:42.056903 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e0aed7c9dcd417d82dac37a8d010373" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:42.057055 20090 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.075344 20122 raft_consensus.cc:3060] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:42.062397 20119 raft_consensus.cc:2468] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 3.
I20260317 16:18:41.997606 20117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:41.894579 21628 consensus_queue.cc:1059] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.139s
I20260317 16:18:41.996542 21470 raft_consensus.cc:697] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
W20260317 16:18:42.088838 20015 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 53887.4us
W20260317 16:18:42.046166 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.064610 21602 raft_consensus.cc:2749] T edd06ee443fc4680b217c7dcbc1f640a P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:42.043021 21652 consensus_queue.cc:1059] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 1, Time since last communication: 0.043s
I20260317 16:18:42.100420 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "951cd4af9fa84309b681fc8e9a6279ff" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
W20260317 16:18:42.069774 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 206390us
I20260317 16:18:42.105372 20236 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:42.106647 20236 leader_election.cc:304] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:42.107156 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.036381 21605 raft_consensus.cc:2804] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 16:18:42.070864 21623 raft_consensus.cc:493] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a8150828b9c4db7b4cdcf2de9f35096)
I20260317 16:18:42.034202 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.036671 21611 raft_consensus.cc:493] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:42.096653 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "352087b67c9f40f4a6fa365aa66b8f13" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:42.118736 20120 raft_consensus.cc:2410] T 352087b67c9f40f4a6fa365aa66b8f13 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260317 16:18:42.119201 20236 leader_election.cc:304] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:42.056789 21587 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:42.051041 20043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd397002573f4a5ba2091161eaea1a47" candidate_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "9a3afb098d6a45f0b405566771a1157b"
I20260317 16:18:42.138890 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a97b610497d410fb77dc8d4e5aca57e" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.140161 20120 raft_consensus.cc:2468] T 9a97b610497d410fb77dc8d4e5aca57e P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 1.
W20260317 16:18:42.052104 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 151509us
W20260317 16:18:42.114156 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.050346 21673 consensus_queue.cc:260] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.071792 21657 consensus_queue.cc:1048] T 65896e94a12d4953b9bbfff15ac73867 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260317 16:18:42.133488 20122 raft_consensus.cc:2468] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:42.100407 21423 raft_consensus.cc:2804] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 16:18:42.126377 20049 raft_consensus.cc:2468] T c902369cf6954cb695208719bf353017 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 6.
I20260317 16:18:42.117003 21611 raft_consensus.cc:3060] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:42.113780 21623 raft_consensus.cc:3060] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Advancing to term 5
I20260317 16:18:42.127310 20042 raft_consensus.cc:2468] T fb492ec38d074e94994a3a2a9e245bac P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 4.
W20260317 16:18:42.163341 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 44266us
I20260317 16:18:42.128505 21605 raft_consensus.cc:697] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 LEADER]: Becoming Leader. State: Replica: 7a8150828b9c4db7b4cdcf2de9f35096, State: Running, Role: LEADER
I20260317 16:18:42.173516 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb3cfd29098d4ab9a179a3d07ebb75ee" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:42.098881 21633 consensus_queue.cc:1048] T 3324f667758c47ca8781d7f7b7822af8 P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260317 16:18:42.104384 21654 raft_consensus.cc:515] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.172005 21606 leader_election.cc:290] T 951cd4af9fa84309b681fc8e9a6279ff P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:42.147286 21651 raft_consensus.cc:2749] T 091c367bf7d84c61b2a329d98d708cc8 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 16:18:42.178386 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a2894e5c7f64210ba697507a2823f16" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:42.331804 20199 raft_consensus.cc:2393] T 3a2894e5c7f64210ba697507a2823f16 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 3: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:42.180366 20119 raft_consensus.cc:2468] T 951cd4af9fa84309b681fc8e9a6279ff P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 4.
W20260317 16:18:42.190740 20088 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 134174us
I20260317 16:18:42.255689 20236 leader_election.cc:304] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:42.332955 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 77348.2us
I20260317 16:18:42.333149 20088 leader_election.cc:304] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:42.333810 20236 leader_election.cc:304] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:42.186702 21601 consensus_queue.cc:237] T 81baa6e6b5864f69b8da5c3418aea90a P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.177698 21587 leader_election.cc:290] T 3a2894e5c7f64210ba697507a2823f16 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:42.167968 21455 raft_consensus.cc:3060] T 7ef41a8cfc184979b0e3fc5d3b370bcd P d8ed4b12dc654776aa7e64c0fa224321 [term 0 FOLLOWER]: Advancing to term 1
I20260317 16:18:42.167568 21242 raft_consensus.cc:2764] T c198f5201a944b26bd4d2f870953e117 P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260317 16:18:42.148797 21626 consensus_queue.cc:1059] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20260317 16:18:42.190943 21470 consensus_queue.cc:237] T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.351334 21626 consensus_queue.cc:1243] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer 9a3afb098d6a45f0b405566771a1157b log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260317 16:18:42.262996 21423 raft_consensus.cc:493] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:42.355182 21423 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:42.355317 21333 raft_consensus.cc:2749] T 43d7dcb7b21548889a850700415701a9 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:42.151489 21690 raft_consensus.cc:2804] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260317 16:18:42.358930 21690 raft_consensus.cc:493] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 16:18:42.359320 21690 raft_consensus.cc:3060] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Advancing to term 7
W20260317 16:18:42.177974 21685 log.cc:927] Time spent T d7faa93fffc94522a88d5c47721186bd P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260317 16:18:42.335000 20088 leader_election.cc:304] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:42.342327 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b302990b671946b29b6e2fcf85413383" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.177054 20118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49737061a2c940b9982aa52ccdf66e90" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:42.364377 21333 raft_consensus.cc:2749] T 1234018d42ac449782301c404118e5bb P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 16:18:42.364595 20118 raft_consensus.cc:3060] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:42.364794 21605 consensus_queue.cc:237] T fb492ec38d074e94994a3a2a9e245bac P 7a8150828b9c4db7b4cdcf2de9f35096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:42.372393 21673 raft_consensus.cc:3060] T 091c367bf7d84c61b2a329d98d708cc8 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:42.372753 21333 consensus_queue.cc:1059] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 16:18:42.374133 21333 raft_consensus.cc:3055] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:42.348930 20119 raft_consensus.cc:1240] T e98310d917124048971895e5cd5c71f1 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260317 16:18:42.174957 21681 log.cc:927] Time spent T 7ef41a8cfc184979b0e3fc5d3b370bcd P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260317 16:18:42.262399 21472 raft_consensus.cc:697] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
W20260317 16:18:42.355202 21687 log.cc:927] Time spent T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
W20260317 16:18:42.392925 20090 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 335912us
I20260317 16:18:42.393992 20090 leader_election.cc:304] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:42.381348 20269 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.012s sys 0.005s
I20260317 16:18:42.381808 20124 raft_consensus.cc:2393] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8ed4b12dc654776aa7e64c0fa224321 in current term 7: Already voted for candidate 795c49e9595f45bdb8d6b568498abf1f in this term.
I20260317 16:18:42.374533 21333 raft_consensus.cc:740] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:42.178423 20047 raft_consensus.cc:2468] T c198f5201a944b26bd4d2f870953e117 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 1.
W20260317 16:18:42.403879 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 439996us
I20260317 16:18:42.396171 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ea78aa57aaf4f86b53cd07ab29faa49" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.408757 20119 raft_consensus.cc:2468] T 1ea78aa57aaf4f86b53cd07ab29faa49 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 3.
W20260317 16:18:42.409132 20016 leader_election.cc:336] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Timed out: RequestConsensusVote RPC to 127.16.123.194:35679 timed out after 1.608s (SENT)
W20260317 16:18:42.396498 20269 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s user 0.012s sys 0.005s
W20260317 16:18:42.393810 21691 log.cc:927] Time spent T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260317 16:18:42.419034 20119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3324f667758c47ca8781d7f7b7822af8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.420219 20119 raft_consensus.cc:2410] T 3324f667758c47ca8781d7f7b7822af8 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260317 16:18:42.422907 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 386828us
I20260317 16:18:42.421061 20016 leader_election.cc:304] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:42.430279 20016 leader_election.cc:336] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Timed out: RequestConsensusVote RPC to 127.16.123.194:35679 timed out after 1.686s (SENT)
I20260317 16:18:42.430876 20016 leader_election.cc:304] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
W20260317 16:18:42.434986 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.437348 20270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021523da58cc4c16babb94f5b4b5cee8" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321" is_pre_election: true
I20260317 16:18:42.396956 21651 raft_consensus.cc:2749] T aa7ec7cb7aad43e8950899557d4022f7 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 16:18:42.441892 20016 leader_election.cc:336] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679): Timed out: RequestConsensusVote RPC to 127.16.123.194:35679 timed out after 1.754s (SENT)
I20260317 16:18:42.444144 20016 leader_election.cc:304] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b; no voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:42.438148 20270 raft_consensus.cc:2468] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 5.
I20260317 16:18:42.447006 20119 raft_consensus.cc:3055] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
W20260317 16:18:42.448868 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 301614us
I20260317 16:18:42.449348 20119 raft_consensus.cc:740] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:42.404888 21574 raft_consensus.cc:2749] T 81baa6e6b5864f69b8da5c3418aea90a P d8ed4b12dc654776aa7e64c0fa224321 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 16:18:42.450469 21592 raft_consensus.cc:2749] T 352087b67c9f40f4a6fa365aa66b8f13 P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260317 16:18:42.451238 20119 consensus_queue.cc:260] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:42.454738 21592 raft_consensus.cc:2749] T 951cd4af9fa84309b681fc8e9a6279ff P 9a3afb098d6a45f0b405566771a1157b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 16:18:42.456815 21574 consensus_queue.cc:1059] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260317 16:18:42.458194 21574 consensus_queue.cc:1059] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260317 16:18:42.459929 21488 consensus_meta.cc:287] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.633s user 0.003s sys 0.006s
I20260317 16:18:42.473562 21488 raft_consensus.cc:515] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.475719 21425 raft_consensus.cc:493] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 16:18:42.476575 21566 consensus_meta.cc:287] T 49737061a2c940b9982aa52ccdf66e90 P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.658s user 0.004s sys 0.007s
I20260317 16:18:42.477293 21333 consensus_queue.cc:260] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.478937 21333 raft_consensus.cc:3060] T 4ff46c37d04d446981ad8728f73c6e1f P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:42.461122 21654 leader_election.cc:290] T 062108d003c14795847b5db10bbabd0f P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 7 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:42.467458 20119 raft_consensus.cc:3060] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:42.467758 20124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a663a988cc754c24b227bb9bc85540db" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.468194 20121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65896e94a12d4953b9bbfff15ac73867" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.468264 21472 consensus_queue.cc:237] T 83e3771d4d494b8caa5d647d8073666c P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
W20260317 16:18:42.482890 20013 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.436568 21583 raft_consensus.cc:2749] T 062108d003c14795847b5db10bbabd0f P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260317 16:18:42.471305 21658 raft_consensus.cc:493] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:42.471676 20122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.476387 21425 raft_consensus.cc:515] T 3a2894e5c7f64210ba697507a2823f16 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:42.477339 20116 raft_consensus.cc:1240] T a07475a709344f5e812eea08aa4113cd P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260317 16:18:42.471392 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a07475a709344f5e812eea08aa4113cd" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f" is_pre_election: true
I20260317 16:18:42.461549 21604 raft_consensus.cc:3055] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:42.487599 20091 leader_election.cc:304] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.490823 20119 raft_consensus.cc:1275] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Refusing update from remote peer 9a3afb098d6a45f0b405566771a1157b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
W20260317 16:18:42.499001 20239 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 46372.7us
W20260317 16:18:42.500415 20090 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:42.491780 21687 log.cc:927] Time spent T e98310d917124048971895e5cd5c71f1 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260317 16:18:42.501134 20119 pending_rounds.cc:85] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: Aborting all ops after (but not including) 0
I20260317 16:18:42.510465 20091 leader_election.cc:304] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.516361 20119 pending_rounds.cc:107] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260317 16:18:42.516531 20091 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.507349 21672 consensus_queue.cc:1059] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260317 16:18:42.520857 21587 raft_consensus.cc:493] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 795c49e9595f45bdb8d6b568498abf1f)
I20260317 16:18:42.521585 21587 raft_consensus.cc:515] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:42.523921 21587 leader_election.cc:290] T 6e7d5e1825514e7abdeac801ec091d2c P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:42.524773 21695 raft_consensus.cc:2749] T ead98604210744a2b26b15fb54b5ea29 P 795c49e9595f45bdb8d6b568498abf1f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 16:18:42.525023 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7d5e1825514e7abdeac801ec091d2c" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:42.527799 20091 leader_election.cc:304] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096
I20260317 16:18:42.527451 21657 raft_consensus.cc:2749] T 9e6529a0cb514111b1bfd58537d255d2 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260317 16:18:42.529513 21695 consensus_queue.cc:1059] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 16:18:42.514958 21488 leader_election.cc:290] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 3 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:42.535279 21704 consensus_queue.cc:1048] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260317 16:18:42.535562 21488 consensus_queue.cc:1048] T d4f0e7842cc94d539a8537a2b131711b P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260317 16:18:42.535959 21623 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.536820 21672 raft_consensus.cc:3055] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 16:18:42.537245 21672 raft_consensus.cc:740] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:42.537097 20117 raft_consensus.cc:2468] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in term 5.
I20260317 16:18:42.509986 21592 raft_consensus.cc:493] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:42.538555 21592 raft_consensus.cc:515] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.540678 21592 leader_election.cc:290] T 8720c3d1b3994abeba1e121c1cf99275 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:42.541391 20199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8720c3d1b3994abeba1e121c1cf99275" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" is_pre_election: true
I20260317 16:18:42.500686 21658 raft_consensus.cc:515] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.546087 21658 leader_election.cc:290] T 4dfc403ac89c4d339adb26bfd789d1a2 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
I20260317 16:18:42.500383 21604 raft_consensus.cc:740] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:42.547556 21604 consensus_queue.cc:260] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:42.549527 21604 raft_consensus.cc:3060] T a29e5bafc1d14cf4932e2a0489026dd2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:42.552116 20239 leader_election.cc:304] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:42.528901 20192 raft_consensus.cc:2393] T 6e7d5e1825514e7abdeac801ec091d2c P 7a8150828b9c4db7b4cdcf2de9f35096 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b in current term 4: Already voted for candidate 7a8150828b9c4db7b4cdcf2de9f35096 in this term.
I20260317 16:18:42.509548 21626 raft_consensus.cc:2749] T e8fd5a883fbb4ae1bcc76cb626ccb7f2 P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:42.516799 20119 raft_consensus.cc:2889] T fb3cfd29098d4ab9a179a3d07ebb75ee P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 16:18:42.538100 21672 consensus_queue.cc:260] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
W20260317 16:18:42.586881 20236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.193:40319, user_credentials={real_user=slave}} blocked reactor thread for 253158us
W20260317 16:18:42.555498 21647 consensus_meta.cc:287] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.696s user 0.007s sys 0.005s
I20260317 16:18:42.553298 21566 raft_consensus.cc:2749] T 4dfc403ac89c4d339adb26bfd789d1a2 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 16:18:42.538122 21623 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 5 election: Requested vote from peers 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885), d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213)
I20260317 16:18:42.595494 20236 leader_election.cc:304] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:42.589251 20014 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:42.596571 21602 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260317 16:18:42.591331 21657 raft_consensus.cc:2804] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 16:18:42.612426 20236 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a3afb098d6a45f0b405566771a1157b, d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f
W20260317 16:18:42.610661 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 184517us
I20260317 16:18:42.556826 21467 consensus_queue.cc:1048] T 5875427d66a54f92898d04d79d1bc6de P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260317 16:18:42.621187 20236 leader_election.cc:304] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
W20260317 16:18:42.622100 20165 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.010s sys 0.014s
W20260317 16:18:42.627414 20165 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.010s sys 0.014s
W20260317 16:18:42.628258 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 595474us
I20260317 16:18:42.630091 20165 leader_election.cc:304] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:42.630923 21579 raft_consensus.cc:2804] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 16:18:42.631337 21579 raft_consensus.cc:493] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a3afb098d6a45f0b405566771a1157b)
I20260317 16:18:42.631740 21579 raft_consensus.cc:3060] T c963e7aa6d46472691e7e0de834008ab P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:42.632308 21652 raft_consensus.cc:3055] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Stepping down as leader of term 3
I20260317 16:18:42.632720 21652 raft_consensus.cc:740] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:42.633116 20091 leader_election.cc:304] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321; no voters:
I20260317 16:18:42.633955 21602 raft_consensus.cc:2804] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 16:18:42.634527 21602 raft_consensus.cc:697] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Leader. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:42.626883 21703 raft_consensus.cc:2804] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 16:18:42.633509 21652 consensus_queue.cc:260] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:42.624511 20236 leader_election.cc:304] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8ed4b12dc654776aa7e64c0fa224321; no voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b
I20260317 16:18:42.584836 21690 raft_consensus.cc:515] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.622995 21467 raft_consensus.cc:2749] T 60cb13d3aec54fd58f4a4821692ab2b3 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260317 16:18:42.641772 21607 consensus_meta.cc:287] T edd06ee443fc4680b217c7dcbc1f640a P d8ed4b12dc654776aa7e64c0fa224321: Time spent flushing consensus metadata: real 0.786s user 0.012s sys 0.003s
I20260317 16:18:42.599362 21647 raft_consensus.cc:515] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.638931 21692 raft_consensus.cc:2749] T b444cc548a6940e98033a9a247164c51 P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 16:18:42.614656 21657 raft_consensus.cc:493] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8ed4b12dc654776aa7e64c0fa224321)
I20260317 16:18:42.595685 21654 consensus_queue.cc:1059] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.023s
W20260317 16:18:42.617821 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 16:18:42.615641 21470 raft_consensus.cc:2749] T 021523da58cc4c16babb94f5b4b5cee8 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:42.641772 21574 raft_consensus.cc:2749] T b302990b671946b29b6e2fcf85413383 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 16:18:42.658752 20120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795c49e9595f45bdb8d6b568498abf1f"
I20260317 16:18:42.637234 21672 raft_consensus.cc:3060] T 9e6529a0cb514111b1bfd58537d255d2 P d8ed4b12dc654776aa7e64c0fa224321 [term 5 FOLLOWER]: Advancing to term 6
I20260317 16:18:42.649641 20116 raft_consensus.cc:1240] T 1e0aed7c9dcd417d82dac37a8d010373 P 795c49e9595f45bdb8d6b568498abf1f [term 5 FOLLOWER]: Rejecting Update request from peer d8ed4b12dc654776aa7e64c0fa224321 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260317 16:18:42.644377 21703 raft_consensus.cc:697] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [term 2 LEADER]: Becoming Leader. State: Replica: d8ed4b12dc654776aa7e64c0fa224321, State: Running, Role: LEADER
I20260317 16:18:42.642414 21607 raft_consensus.cc:493] T 41e4ff5d2d614a199e8e8a314b43bb7c P d8ed4b12dc654776aa7e64c0fa224321 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 16:18:42.635699 20165 leader_election.cc:304] T 80b48fd655d14d649c65e99261b01ca9 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:42.645294 20190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a510b634dc4a808c9f7af78ba4cd7e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
W20260317 16:18:42.685704 20165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 50088.8us
I20260317 16:18:42.685843 20190 raft_consensus.cc:3060] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 2 FOLLOWER]: Advancing to term 3
I20260317 16:18:42.686839 20165 leader_election.cc:304] T 9e6529a0cb514111b1bfd58537d255d2 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096; no voters: 795c49e9595f45bdb8d6b568498abf1f, d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.651746 21657 raft_consensus.cc:3060] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 FOLLOWER]: Advancing to term 2
I20260317 16:18:42.656683 21701 raft_consensus.cc:3055] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Stepping down as leader of term 1
I20260317 16:18:42.688771 20165 leader_election.cc:304] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:42.641151 21611 raft_consensus.cc:515] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } }
I20260317 16:18:42.643126 21423 raft_consensus.cc:515] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } }
I20260317 16:18:42.692523 20165 leader_election.cc:304] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 7a8150828b9c4db7b4cdcf2de9f35096; no voters:
I20260317 16:18:42.665881 21652 raft_consensus.cc:3060] T 6e7d5e1825514e7abdeac801ec091d2c P 795c49e9595f45bdb8d6b568498abf1f [term 3 FOLLOWER]: Advancing to term 4
W20260317 16:18:42.690022 20089 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260317 16:18:42.695111 20016 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.123.194:35679, user_credentials={real_user=slave}} blocked reactor thread for 253264us
I20260317 16:18:42.704988 20016 leader_election.cc:304] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.707724 20264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:42.708861 20269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "062108d003c14795847b5db10bbabd0f" candidate_uuid: "795c49e9595f45bdb8d6b568498abf1f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8ed4b12dc654776aa7e64c0fa224321"
I20260317 16:18:42.709223 20193 raft_consensus.cc:1240] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Rejecting Update request from peer 9a3afb098d6a45f0b405566771a1157b for earlier term 5. Current term is 6. Ops: []
I20260317 16:18:42.709501 20269 raft_consensus.cc:3060] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Advancing to term 7
I20260317 16:18:42.694173 20266 raft_consensus.cc:2368] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 795c49e9595f45bdb8d6b568498abf1f for earlier term 5. Current term is 6.
I20260317 16:18:42.792069 21611 leader_election.cc:290] T a9f666d8f908427cb4523d57a45eb2b3 P 7a8150828b9c4db7b4cdcf2de9f35096 [CANDIDATE]: Term 6 election: Requested vote from peers d8ed4b12dc654776aa7e64c0fa224321 (127.16.123.196:43213), 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319)
I20260317 16:18:42.799290 21690 leader_election.cc:290] T 352087b67c9f40f4a6fa365aa66b8f13 P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 7 election: Requested vote from peers 9a3afb098d6a45f0b405566771a1157b (127.16.123.193:40319), 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679)
@ 0x7fea32ae7c87 __libc_start_main at ??:0
I20260317 16:18:42.801919 21657 raft_consensus.cc:515] T fca192bf1e8848e2a2dac227f537c9e2 P 795c49e9595f45bdb8d6b568498abf1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } } peers { permanent_uuid: "7a8150828b9c4db7b4cdcf2de9f35096" member_type: VOTER last_known_addr { host: "127.16.123.195" port: 36885 } } peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } }
I20260317 16:18:42.802835 20193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d13fb540668437094ca4190aa5dd7ec" candidate_uuid: "9a3afb098d6a45f0b405566771a1157b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:42.803572 20193 raft_consensus.cc:3060] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Advancing to term 6
W20260317 16:18:42.811347 20238 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.123.195:36885, user_credentials={real_user=slave}} blocked reactor thread for 102814us
I20260317 16:18:42.818146 20193 raft_consensus.cc:2468] T 6d13fb540668437094ca4190aa5dd7ec P 7a8150828b9c4db7b4cdcf2de9f35096 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3afb098d6a45f0b405566771a1157b in term 6.
I20260317 16:18:42.710321 20016 leader_election.cc:304] T 9a97b610497d410fb77dc8d4e5aca57e P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 795c49e9595f45bdb8d6b568498abf1f, 9a3afb098d6a45f0b405566771a1157b; no voters: d8ed4b12dc654776aa7e64c0fa224321
I20260317 16:18:42.710541 20270 raft_consensus.cc:1240] T 34a510b634dc4a808c9f7af78ba4cd7e P d8ed4b12dc654776aa7e64c0fa224321 [term 3 FOLLOWER]: Rejecting Update request from peer 795c49e9595f45bdb8d6b568498abf1f for earlier term 2. Current term is 3. Ops: []
W20260317 16:18:42.713130 21710 log.cc:927] Time spent T 0709d1e3fc7349f2b93ed3e602c1a723 P d8ed4b12dc654776aa7e64c0fa224321: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260317 16:18:42.828397 20192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef04e628857445e2a7a7cd12a684908e" candidate_uuid: "d8ed4b12dc654776aa7e64c0fa224321" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "7a8150828b9c4db7b4cdcf2de9f35096"
I20260317 16:18:42.714990 21658 raft_consensus.cc:2764] T f26f14718ce8437ab76d99156bfa1fa3 P 9a3afb098d6a45f0b405566771a1157b [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260317 16:18:42.719290 21695 raft_consensus.cc:3055] T b302990b671946b29b6e2fcf85413383 P 795c49e9595f45bdb8d6b568498abf1f [term 4 LEADER]: Stepping down as leader of term 4
I20260317 16:18:42.706069 21633 raft_consensus.cc:2804] T 1e0aed7c9dcd417d82dac37a8d010373 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 5 FOLLOWER]: Leader election won for term 5
W20260317 16:18:42.723253 21706 log.cc:927] Time spent T 65896e94a12d4953b9bbfff15ac73867 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260317 16:18:42.724496 21703 consensus_queue.cc:237] T f26f14718ce8437ab76d99156bfa1fa3 P d8ed4b12dc654776aa7e64c0fa224321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.729168 20190 raft_consensus.cc:2468] T 34a510b634dc4a808c9f7af78ba4cd7e P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ed4b12dc654776aa7e64c0fa224321 in term 3.
I20260317 16:18:42.820259 20015 leader_election.cc:304] T 6d13fb540668437094ca4190aa5dd7ec P 9a3afb098d6a45f0b405566771a1157b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8150828b9c4db7b4cdcf2de9f35096, 9a3afb098d6a45f0b405566771a1157b; no voters:
I20260317 16:18:42.730408 20269 raft_consensus.cc:2468] T 062108d003c14795847b5db10bbabd0f P d8ed4b12dc654776aa7e64c0fa224321 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795c49e9595f45bdb8d6b568498abf1f in term 7.
W20260317 16:18:42.731360 21711 log.cc:927] Time spent T fd3be07aef26463ba2eaa9c72c6fa094 P 7a8150828b9c4db7b4cdcf2de9f35096: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260317 16:18:42.693065 21488 raft_consensus.cc:3055] T 1e0aed7c9dcd417d82dac37a8d010373 P d8ed4b12dc654776aa7e64c0fa224321 [term 4 LEADER]: Stepping down as leader of term 4
@ 0x29847a _start at ??:?
I20260317 16:18:42.732190 21583 consensus_queue.cc:1243] T c963e7aa6d46472691e7e0de834008ab P 9a3afb098d6a45f0b405566771a1157b [LEADER]: Peer 795c49e9595f45bdb8d6b568498abf1f log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260317 16:18:42.691179 21701 raft_consensus.cc:740] T 4dfc403ac89c4d339adb26bfd789d1a2 P 795c49e9595f45bdb8d6b568498abf1f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 795c49e9595f45bdb8d6b568498abf1f, State: Running, Role: LEADER
I20260317 16:18:42.733598 21647 leader_election.cc:290] T ef04e628857445e2a7a7cd12a684908e P d8ed4b12dc654776aa7e64c0fa224321 [CANDIDATE]: Term 6 election: Requested vote from peers 795c49e9595f45bdb8d6b568498abf1f (127.16.123.194:35679), 7a8150828b9c4db7b4cdcf2de9f35096 (127.16.123.195:36885)
I20260317 16:18:42.733755 20118 raft_consensus.cc:2410] T 49737061a2c940b9982aa52ccdf66e90 P 795c49e9595f45bdb8d6b568498abf1f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a3afb098d6a45f0b405566771a1157b for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260317 16:18:42.694178 21602 consensus_queue.cc:237] T c198f5201a944b26bd4d2f870953e117 P 795c49e9595f45bdb8d6b568498abf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ed4b12dc654776aa7e64c0fa224321" member_type: VOTER last_known_addr { host: "127.16.123.196" port: 43213 } } peers { permanent_uuid: "9a3afb098d6a45f0b405566771a1157b" member_type: VOTER last_known_addr { host: "127.16.123.193" port: 40319 } } peers { permanent_uuid: "795c49e9595f45bdb8d6b568498abf1f" member_type: VOTER last_known_addr { host: "127.16.123.194" port: 35679 } }
I20260317 16:18:42.694195 21458 raft_consensus.cc:2804] T 091c367bf7d84c61b2a329d98d708cc8 P 7a8150828b9c4db7b4cdcf2de9f35096 [term 3 FOLLOWER]: Leader election won for term 3